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AN OVERVIEW OF THE OREGON STUDIES IN EDUCATIONAL RDD&E 



In the spring of 1970 the Training Branch of the U.S. 
Office of Education. National Center for Educational 
Research and Development, announced a plan to effect 
change in the preparatio' of educational RDDAE person- 
nel. Two factors led to the announcement. The underlying 
factor was the rather dramatic emergence in the past decade 
of development, diffusion, and evaluation activiUes as 
vehicles for educational improvement, and the attending 
need for qualifled personnel to carry them out. The 
precipitating factor, however, was evidence that in spite of 
an investment of approximately 30 million dollars by the 
Federal Government to help training programs become 
more responsive to the personnel needs created by these 
new actiWties, essentially the same number and kind of 
personnel were being prepared in 1970 as in 1965. 

The plan for change reflected a strategy that can best be 
described as “beginning at the beginning.” It incorporated 
three intenelated lines of activity: the creation of a 
conceptual and empirical base on which to build functional 
training programa; the design of more effective and efficient 
approaches to training; and the development of instruc- 
tional materials that reflect desired changes in both content 
and procedure. The propositions on winch the plan rested 
were straightforward: (g) little was known about edu- 

cational development, diffusion and evaluation activities, or 
how they related to educational reaeai'ch; (b) even leu was 
known about the training of personnel to carry out such 
activities; and (c) until both of these conditions were 
remedied the likelihood of designing effect”* and efficient 
programs to prepare personnel to carry them out was slight. 
Die plan as a whole was coordinated so that the various 
activities within it would be developed with sensitivity to 
each other, and so that they would come together in 
completed fashion at approxinutely the nme point in time. 
(For additional details on the plan for change see Gupter I 
in Volume I of the series reporting the Oregon Studies.) 

The Oregon Studies, carried out by the Teaching 
Research Division of the Or^n State Syitem of Higher 
Education, were to contribute in a begiruiing way to the 
conceptual and empirical base called for in the plan. As 
such they were to produce five products: a collection of 
detailed “case study” descriptioiu of projects that illus- 
trated exemplary RIXlftE activities within various edu- 
cational contexts; a reliable, economically feasible method- 
ology by which to collect the data needed to prepare the 
case studies; a conceptual system or framework for viewing 
the doiw.in of educational RDD&E t!uit could be used as a 
guide to the classes of data to be attended to in the case 
studies; cross-project analyses that highlighted the simi- 




larities and differences observed in the projects described, 
and that tested in rudimentary faihion the adequacy of the 
conceptual framework underlying those observations; and a 
compendium of the existing literature that pertained to 
either the nature of or the interactions between activities 
labeled educational research, development, diffusion and 
evaluation. These products are report^ in five volumes: 



Volume I. Summary Report (with Technical 
Appendices) 

Volume II. The Literature of Educational RDD&E 
Part One (Research, Evaluation, and 
Development) 

Part Two (Diffusion & Combinations of 
RDD&E) 

Volume III. Conceptual Frameworks for Viewing 
Educational RDD&E 



Volume IV. Profiles of Exemplary Projects in 
Educational WD&E 
Part One (Research and Evaluation) 
Part Two (Development) 

Part Three (Diffusion) 

Volume V. A Methodology for the Study of 
Educational RDD&E 



Each volume in the series reporting the Studies has been 
deigned to stand alone, but because each volume reports a 
different product, and each product can be understood 
fully only in relation to the other products, two “reader’s 
guites” to. the series have been prepared. Die first involves 
brief summaries or abstracts of the contents of each of the 
five volumes in the series. These appear on the inside of the 
back cover of the volume, and are intended to serve as a 
guide or overview to the series as a whole. A more detailed 
guide is provided by Volume I. In addition ‘3 serving as a 
general summary of the Studies, it contains deKriptions of 
the developmental histories of the products reported in the 
vious volumes, the relationships that exist between .them, 
ai.d the manner in which they have interacted over time. 
Accordingly, for the reader who wishes to determine 
quickly what each of the five volumes in the aeries contains, 
turn to the inside of the back cover of the volume; for the 
reader who wishes to understand how the volumes relate to 
one another, follow that by reading Volume I. 
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ALSTRACT 



This is one of five volumes reporting the results of the Oregon 
Studies in educational research, development, diffusion, and evaluation 
(educational RDD6E). It conti*ins 20 case study profiles of educational 
RDD&E pvojects, and as such constitutes the data base for the Oregon 
Studies. The Volume is bound in Three parts. Part One contains profiles 
of five research and three evaluation projects; Part Two contains profiles 
of seven development projects; and Part Three contains profiles of five 
diffusion projects. Each part within the volume contains information 
that describes the development of the profiles, how to read the profiles, 
and a glossary of common profile terms. Each profile contains three 
sets of data: (^) descriptors of general project characteristics, c.g., 

objectives, timelines, organizational structures, and project **dynamics;** 
do descriptors of personnel working wAthin projects. Including back- 
ground of training, work experience, apd job role definition; and (c) 
descriptors of the work requirements within a project. Work requirement 
data include descriptions of the outputs that derive from a project, the 
standards held for those outputs, the operations required to produce 
outputs to the standards specified, and the knowledges , skills , and 
sensitivities drawn upon to carry out project operations. Nine hundred 
and sixty-two outputs of work effort were identified in the 20 projects. 
Two hundred and ninety-eight of these were analyzed for their work 
requirements. From thin analysis 1148 descriptions of standards, 

3722 descriptions of tasks, and 2974 descriptions of knowledges, skills, 
and sensitivities were obtfiined. One hundred and thirty-four pro- 
fessional persons were interviewed in collecting these data. The profiles 
are discussed in the preface to the volume from the point of view of 
their utility as scientific and training documents. 
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PREFACE 



Tlie present volume contains descriptive profiles of 20 educational 
research, development, diffusion, and evaluation (cduralional RDD6E) 
projects. The volume is bound in three parts. Part One contains pro- 
files of 5 research and 3 evaluation projects; Part Two contains profiles 
of 7 development projects; and Part Three contains profiles of 5 dif- 
fusion projects. In addition, each part within the voJume contains 
information that describes the development of the profiles, information 
that serves as a guide to reading the profiles, and a glossary of common 
profile terms. In combination, these materials should permit a reader 
to study the profiles with sensibility and understanding# 

Each profile attempts to portray the essential characteristics of 
the project it describes and the realities of work requirements within 
it. Toward these ends, each profile describes: (a) the general charac- 

teristics of a project, e.g., objectives, timelines, organizational 
structures, and project "dynamics;" (^) the characteristics of personnel 
working within a project, including background of training, work experi- 
ence, and Job role definitions; and (£) the work requirements within a 
project . 

The central data reported in a profile deals with project work 
requirements. In this regard, each profile describes the outputs of 
work effort, the standards established for those outputs, the operations 
required to produce outputs to specified standards, and the knowledges , 
skills , and sensitivities needed to carry out those operations. An over- 
view of the data sets used to describe these variables and their inter- 
dependencies is provided in the re**der*s guide to the profiles. The 
rationale for and a full description of the data iets used is provided 
in Chapter 4 of Volume I of the series of volumes reporting the Oregon 
Studies . 

The profiles were designed to serve the purposes of both science 
and training. In support of science the profiles serve three functions: 
(£) the careful description of phenomena of interest; (h) the develop- 
ment of a methodology by which to carry ov 5 t such description; and 
(c) the development of a data base that permits parameter identification 
and comparative analyses. In support of training the profiles se*rve 
two functions: (a) they provide a means of gaining insight into the 

nature of and work requirements within individual educational RDD&E 
projects; and (b) they provide a means of gaining insight into the 
nature of and work requirements within the domain of educational RDD&E 
as a whole. Because these various concerns have combined to make the 
profiles as they are, each will be discussed briefly. 

PROFILES AS BASIC SCIENCE DESCRIPTIONS. Individually and collec- 
tively the profiles provide accurate, reliable, and relatively exhaus- 
tive descriptions of ongoing RDD&E activities at the project level# 

All projects described are illustrative of the kinds of RDD&E activities 
likely to he funded in the decade ahead. The rationale for obtaining 
such descriptions involved a series of related propositions: (^) research, 

development, diffusion, and evaluation activities have served as powerful 

I . ' 
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problem .solving tooiis in i wide range of manVs LMicicavurs, e.g., medicisie. 
agriculture# and industry, but as vet their systern.itit application witliin 
the context of education has been limited; (^)to have applicability within 
the context of education RDD&E activities must be adapted to fit partic- 
ular demands of education; (£> to effectively bring about such adaptation, 
the demands of RDD 61 E within education must be understood; (d) at the time 
that the Oregon Studies W(fre undertaken little was known about cdurational 
development, diffujiion, atid evaluation activities, about how such activi- 
ties related to educational research, or about how any or all activities 
related to the improvement of education; and (£) in order to understand 
matters not understood it is wis'j to begin by describing them in detail. 

The rationale for reporting such descriptions in case profile format was 
less complex: It invited a more detailed description of project charac- 

teristics and activities than might otherwise be provided. Tliis was 
assumed to bo true for both the identification of the variables to be 
attended to in describing projects and L#ie exploration of the inrer- 
actions of those variables. 

PROFILES AS METHODOLOGICAL PROVING GROUND. The decision to describe 
educational RDD&E projects in case profile terms required that a method- 
ology be developed that would generate '*c.ise study" data. The develop- 
ment of su*-h a met!iodology became a primary focus of the Oregon Studic., 
and the preparation of profiles waf, to a large extent, a natural culmina- 
tion of that focus. Two assumptions accompanied tlic emphasis on method- 
ological development: (£) the Oregon Studies represented the first in 

a series of empirical studies to be undertaken on the nature of educa- 
tional RDD&E; and greater benefits would accrue to education over 
the long term by directing limited resources to the development of strong 
methodology than would accrue had the investment of resources been directed 
to the collection of large amounts of data with a weaker methodology. 

As a proving ground for methodology, the profiles provided a basis 
for making two kinds of judgments: (a) judgment as to the sophistication 

of the methodology, i.e., the extent to which the metliodology generates 
accurate, reliable, and reasonably exhaustive descriptions of educational 
RDD&E activities; and (^) judgment as to the robus tness of the methodology, 
i.e., the extent to which the methodology can be applied to widely 
varying projects with equally productive results. Evidence as to 
sophistication was obtained by submitting completed profiles of projects 
to the directors of those projects for review and approval. In all cases 
the profiles met the criteria of sophistication outlined above (see the 
Notes on the Development of the Profiles for project director evaluations). 
Evidence as to robustness was obtained by applying the methodology to 
the 20 projects described in the present volume. These projects varied 
widely, and it was assumed that if the methodology was Indeed adequate 
in terms of its robus tness each of the 20 projects could be described with 
equal facility. It was also assumed that the data generated in relation 
to each project would be roughly comparable. As will be seen upon reading 
the profiles, those criteria have been met. An overview of the method- 
ology is provided in the reader's Guide to the profiles. A detailed 
description of the methodology, as well as a description of the manner 
in which it evolved, is provided in Volume V of the series of volunes 
reporting the Oregon Studies. 
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TROFILKS AS A DATA BASK FOR I’ARAMKTKK I Dl.NTI F I CATION AND COMPAKy\TIVF 
ANALYSKS. Tlie decision to view Llio ^rofilcK nf; n dntn base for IdenLifvliu’ 
or ’^mapping’’ the parameters of the domain of educational RDD&K ernerRod as 
a logical extension of the two previously discussed profilo t uiict lon‘ 4 . 

Since extensive descriptive data on the nature of educational RDD&K wi*re 
to be made available as a lesult of profile development, and since profiles 
were to be prepared for widely varying projects to test the robustness of 
a methodology, the selection of the projects to be described was approached 
from the point of view tliat they represent a sample o^ tnc projects that 
exist within the domain of educational KDD&K as a whole. Given the small 
number of projects that could be described In case study form with the 
resource base available, and given tlic variability that was to be reflected 
In those projects, no illusions were held about the representativeness 
of the sample that could be drawn. At the same time. It was reasoned 
that if tlie projects to be described sampled at all well the variability 
that existed in projects within the domain, the descriptions of those 
projects would provide at least a beginning base for sketching an "outline 
map" of the parameters of the domain. As an outgrowth of this kind of 
reasoning, it was decided that projects should vary systematically with 
respect to major sources of variability In educational RDD&E projects as 
a whole. Accordingly^ the 20 projects described vary as to focus (research, 
development, diffusion, and evaluation), size (a funding base of less than 
$100,000 per annum, between $100,000 and $250,000 per annum, and over 
$250,000 per annum), and setting (public schools and state departments 
of education, colleges and universities, publiciy fundee laboratories 
and R&D centers, and privately funded R&D centers). A description of 
the procedures followed and criteria used in selecting the 20 projects 
is provided in Chapter 3 of Volume 1 of the series of volumes reporting 
the Oregon Studies. 

As a data base for mapping the domain of educational RDD6i£, the 
profiles actually serve two functions: (£) they provide a basis for 

mapping the parameters of the domain; and (^) they provide a basis for 
mapping the commonalities or central teadencics of the domain. As a 
basis for parameter mapping the profiles constitute an excellent source 
of data. Even though the project sample is small, and the absolute data 
base on which to prepare maps limited, projects have been selected so 
as to insure that they are reasonably representative of the range of 
projects to be found within the domain of educational RDD&E. Thus, the 
range of personnel employed in the 20 projects described, the range of 
project strategies followed, the range of organizational structures used, 
the range of outputs produced, the range of tasks performed, the range of 
standards held, and the range of knowledges, skills, and sensitivities 
drawn upon in their execution can be assumed to be reasonably representa- 
tive of the range of such things to be found within the domain as a whole. 

The technical appendices that accompany Volume I of the series of volumes 
reporting the Studies summarize these data. 

Given the sampling strategy that was followed, it is obvious that 
the profiles constitute a much weaker data base for mapping commonalities 
or central tendencies. Clearly, the sample was drawn to highlight the 
parameters of the domain rather than its central tendencies. Nevertheless, 
the data are amenable to central tendency analyses, and they wore u^.der- 
taken. The "outline maps" presented in Chapters 6, 7, and 8 of Volume I 

V 
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of Llie series of volumes reporting; the Oregon iituilles summarize these 
(lata. 



PKOFILKS AS TKAI'ilNt* AII)S. As tnc most (ietalied descriptions of 
ongoing RDD&M act iv 1 1 le?^' available, it was anticipated tliat the profiles 
could serve a valuable training function. Readers should find, for 
example, that tliey Illustrate the nature of tlie work found within 
educational projects, the i.ature of the tasks Involved in carry- 

ing out that work, the knowledges, skills, and sensit Ivities needed to 
carry it out, the interpersonal and interagency dynamics involved in 
project operation, etc. Such information should be oi value to students 
preparing to enter the field of educational RDD&h, staff wlio Ijavc just 
entered tlie field, or project directors wlio need to provide on the Job 
training. 



PROFILES A:^D cross PROFILE ANALYSES AS A BASIS FOR TRAINING PROGIOVM 
DESIGN. By treating each of the 20 profiles as reliable descriptions of 
**what life is like* within the context of educational RDU&E projects, 
by treating the suramated data as a trustworthy description of the range 
of project activities within the domain as a whole, and by having at 
hand whatever central tendency data that can be gleaned from the compar- 
ative analyses of projects, the designer of training programs should be 
in a p' iitlon to make reasonably informed decisions as t^' what the focus 
ard CO*' tent of those programs should be. In combination these data begin 
to provide the designers of training programs with a sense of the arena 
wi^thin which educational RDD&E personnel must function, and with a sense of what 
has to be done to function effectively within that arena. Chapter 14 of 
Volume I of the series of volumes reporting the Oregon Studies spells 
out some of the implications that derive from these various data sources 
for the design of training programs. 

A wide range of persons have been involved in the preparation of 
the profiles. In fact, nearly all persons involved in the Oregon Studies 
have contributed in one way or another, for essentially all activities 
undertaken within the studies have pointed towards profile production. 

Since other volumes detail the activities that have been related to 
profile development, e.g., the development of the methodology used to 
collect the data reported in the profiles (Volume V) and the develop- 
ment of the conceptual framework that guided the methodology (Volume 
III), the persons involved most directly in those activities need not 
be recognized here. Those who have been most directly involved in 
profile preparation do, however, and the purpose of the following 
paragraphs is to make that recognition public. 



It is proper to acknowledge first those persons in the U.S. Office 
of Education who had the wisdom and courage to insist upon the develop- 
ment of case profiles, and their accompanying methodology, as the 
primary outputs of the Oregon Studies. In this regard the efforts of 
Ms. Cora Beebe and Drs. John Egermeier, Sue Kleini and Paul Messier 
deserve special recognition. So do the efforts of Dr. John Hopkins 
of Indiana Universicy, the U.S. Office of Education’s special consultant 
to the project. The contributions of theseujfive people to the design 
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;in'l Inplrmeti ta^ l('n c. f the case profiles and the supportit.p^ methodology 
h/ivc been of Inestlmahle value. Also deserving of recogtiition la the 
role played in the project by l!S()K project officers. Their willingness 
lo review projects to help in Identifying those that appeared to meet 
the criteria f('r inclusion in the Oregon Studies was clearly beyond 
tlieir estabJlsiied duties. My thanks to all in OSOK whe have given 
so much. 

1 wish to express my thanks also to the directors of the various 
projects for which case profiles were prepared, and to their staffs. 

It Is not easy to give up as mu:h as three i!ays of time when conducting 
a major HDD or \] project, or to release major staft members for as much 
as a day or a day and a half to do other than project work. I’artlcipa- 
tion In the Oregon Studies represented a sizeable Investment of these 
people’s time and energy, and 1 wish to exj)ress my deepest appreciation 
for their willingness to make such an Itnestment. 

Kinally^ I wish to express my thanks to the staff of the Oregon 
Studies who were responsible for data collection, reduction, and profll> 
preparation. Since so many have been Involved, and In so many dlfforcnt 
ways, 1 will simply list names by activity. Thus, the task of refining 
the criteria for project selection. Identifying orojects that met those 
criteria, and making initial contact with those projects relative to 
participation in the study: the team of Mr. Steve Anderson, Mr. Harrell 

Clukey, Dr. Dale Hamreus, and Dr. Jim Nord; the task of making site visits 
for purposes of final project selection: the team c f Dr . Harry Ammerman, 

Dr. Dale Hamreus, and Mr. Greg Thomas; the task of data collection, 
reduction, and initial profile preparation: Mr. Loring Carl, Mr. Norman 

Crowhurst, Mrs. Lee Green, Mr. lierb Hill, Mrs. Jiane Jones, Dr. Rod Myers, 
Dr. Jim Nord, Mr. Dean Piclstick, Mr. Clark Smith, and Mr. Greg Thomas; 
the task of profile editing and refinement: Dr. Harry Ammerman, Mr. 

Loring Carl. Mr. Darrell Clukey, Dr. Kevin Morse, and Mr. Greg Thomas; 
the task of coordinating and scheduling the interview teams: Mr. Greg 

Thomas; the task of interview team training, and the task of administer- 
ing quality control checks on all data reduction: Mr. Loring Carl and 

Mr. Clark Smith; the task of tracking all data from the time it came in 
from the interview teams until it was organized and presr»nted within 
a completed case profile, including the task of editing each profile 
to assure consistency and quality: Mr. Darrell Clukey; the task of 
transferring the reduced data to computer storage, the preparation of 
computer programs for the analysis of the data, and the execution of 
those analyses: Mr. Bill Hickok; the task of overall activity 

coordination: Dr. Harry /Vmmerman. 

My deepest thanks to all for tasks well done. 



H. Del Schalock 

Director of the Oregon Studies 
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NOTES ON niE DEVELOPMENT OF THI- PRO^^ILES 



Fourteen specifications guid^^d the development of the profiles that 
appear in the present volume. Seven of the 14 pertained to the coni out 
of the profiles. 

1. They were to accommodate widely varying data within a standard 
format, that is, a single format was to accommodate data 
emerging from an ^'evolving" case study methodology that was to 
be applied to projects of widely varying characteristics; 

2. They were to convey both the "essential" features of a project 
(as opposed to every possible feature), and the "realities" 

of work within it; 

3. They were to include a description of the context within which 
a project was operating; 

4. They were to include both, but discriminate between, sub- 
jectively and objectively derived data; 

5. They were to include the "raw" data from which categorized 
data emerged; 

6. They were to highlight the training implications that emerged 
from the study of a particular project; and 

7. They were to avoid description of the substantive content of 
a project, except as needed to understand a project within 
the context of the profile. 

Four specifications pertained to the form of the profiles: 

1. ley were to preserve the anonymity of persons within projects; 

2. They were to be candidly written, but without evaluative 
overtones and without reference to outside standards for 
comparative purposes; 

3. To the extent possible, each profile was to make a unique 
contribution to the set of profiles (thus allowing indi- 
vidual differences between profiles with respect to degree 
of emphasis on various classes of data, depth cf detail, 
etc.); and 

4. They wc^re to be readable and understandable by persons just 
entering the field. 

Three specifications pertained to the means by which the profiles were 
prepared : 




1. The profile design, and the linkage of that design to data, 

was lo be such that persons relatively unskilled in professional 
report writing could, without elaborate training, assemble 
and prepare a profile; 

2. Profiles were to be prepared and made available for review 
and/or use as soon as possible after the analysis of a 
project had been completed; and 

3. Profiles were to be approved before publication by the directors 
of the projects described. 

In sum the task of the Oregon Studies was to develop a procedure and 
a format for writing profiles that would display widely differing kinds 
of data from widely differing projects in a manner that would be easily 
understood, and that would allow for comparability across projects while 
retaining the ability to present characteristics idiosyncratic to indi- 
vidual projects. Furthermore the procedure and format were to accomodate 
the variability introduced in data by an "evolving*' methodology, and were 
to be able to be applied by persons with little or no experience in formal 
report writing. The profiles reported in the volume meet or have met 
these specifications . 

Proceduraily , profile design progressed through six identifiable 
stages. The first stage occurred prior to data collection activities, 
and involved the outlining of alternative profile formats for anticipated 
data. These were prepared for conference review in conjunction with the 
first review of the proposed methodology (July 1970). In the second 
stage of development, alternative profile formats were prepared for a 
single project using trial data collected on that project. These were 
prepared for conference review in conjunction with the second review 
of the methodology (October 1970). It was through these two external 
review conferences that most of the specifications relative to the 
development of the profiles emerged. 

The third stage in the evolution of the profiles involved the 
development of a format that accommodated both the specifications that 
had been developed, and the data that were by then emerging from application 
of the methodology. Four profiles were prepared according to this format, 
and submitted for conference review in conjunction with the third external 
review of the methodology. This was held in March 1971, and constituted 
the last formal review of the profile format. In all three of the 
external review sessions, participants included the consultants to the 
Oregon Studies, training program directors, U.S. Office of Education 
personnel, and the authors of the conceptual papers that appear in Volume 
III of the series of volximes reporting the Oregon Studies. 

Following the March review, the profile format went through three 
additional "fine tuning" stages in its development. The first of these 
(Stage 4 in the development of the profile formats) incorporated both 
the recommendations received at the March conference and the subtle 
shifts that occurred in data collection strategy following that conference. 
Six profiles were prepared using this particular format. The next to last 
refinement in format (Stage 5) reflected the final refinement in data 
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collection methodology, and was used in describing the remaining 10 
projects analyzed. The final refinement in format (Stage involved an 
internal review of the total set of profiles from the point of view of 
standardizing terminology, table headings, and category labels. 

Because of the evolution of data collection methodology and profile 
format during the course of the Oregon Studies, it was not possible to 
achieve complete standardization across profiles. The first four profiles 
prepared contained data that were sufficiently different from the data 
reported in the next six, and the data reported in those six were suf- 
ficiently different from that reported in the last 10, Lnat differences 
between the three sets could not be eliminated by the final refinement 
effort. As a consequence, the total collection of profiles reflect 
three recognizably different formats, as well as three slightly different 
data sets. All profiles contain the same basic chapter organization, 
however, and the same major headings within chapters, so differences 
between profile sets are minimal. The GUIDE TO READING THE PROFILES 
has been designed both to introduce the reader to the substantive content 
and organization of the profiles, and to place the differences in profile 
format in perspective. Chapter 4 of Volume I of the series of volumes 
reporting the Oregon Studies traces the implications of profile format 
dif v^erences for cross-project analyses. 

A number of procedures were adopted as guides to the preparation 
of profiles. Profile writers were always members of the data collection 
team and they always knew in advance when they were to serve as writers- 
To insure consistency across writers, chapter titles, major headings 
within chapters, data tables and figures, and data sources were standard- 
ized. During the actual process of preparing the profiles, writers were 
instructed to make use of all record forms, tape recorded interviews, 
and data presentations. Debriefing sessions conducted with the members 
of the data collection team were held to further the writer's under- 
standing of both the project as a whole and the data collected in rela- 
tion to it. 

Profile drafts were given substantive critiques by all members of 
the data collection team, and editorial critiques by at least two other 
Oregon Studies staff. Where extensive revisions were needed, the 
revised drafts were subjected a second time to a complete review and 
critique process. Upon completion, each profile was submitted for 
review and approval to the responsible officer of the project being 
described. The last five profiles submitted to project officers were 
accompanied by a profile rating sheet in order to obtain specific infor- 
mation as to their adequacy. The results of these ratings are summarized 
in Table 1. 
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TABLE 1 



Frequency of Ratings as to Profile Adequacy 

(N = 5) 





Rating Schec 


ule 


Focus of rating 


A 


B 


c 



1. Description of organizational structure 

2. Description of organizational operations, 
interrelationships 

3. Description of the ends being sought by 
the project 

4. Representativeness of all outputs indexed 
(Ch. II) 

5. Representativeness of the outputs analyzed 
(Ch. Ill) 

6. Accuracy of the data presented on outputs 
(Appendix) 

7. Overall representativeness of the Profile 
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Rating Schedule 

Check box A, B, or C as follows: 

A « Representative of a majority of operational concerns. 

B = Representative of only a part of operational concerns 
(concerns of significant proportions omitted) . 

C = Major concerns not covered. 



A GUIDE TO READING THE PROFILES 



Since the profiles are relatively complex documents, and since 
they vary in format (see NOTES on the development of the profiles), a 
guide to their reading has been prepared. The guide is designed to 
orient the reader to (^) the classes of data reported in the profilejj, 
(^) the procedures followed in collecting those data, (c) the trust- 
worthiness of those data, and (^) the manner in which the data have 
been organized within the profiles. [f used in conjunction with the 
NOTES on the development of the profiles and the GLOSSARY of profile 
terms that also appear in the volume, a reader should have no dif- 
ficulty in making his way through the profiles. 



Classes of Data Reported in the Profiles 

As indicated in the preface to the volume three major classes of 
data are reported in each profile: (£) descriptors of general project 

characteristics; (Jb) descriptors of project personnel; and (O descrip- 
tors of project work requirements. Work requirement data are reported 
both in terms of work activities associated with job roles and v/ork 
requirements associated with project outputs. The data sets that 
comprise these various data classes are described briefly in the para- 
graphs that follow. The rationale for and full description of the data 
sets appear in Chapter 4 of Volume I of the series of volumes reporting 
the Oregon Studies. 

Data Sets Used in Describing the 
General Characteristics of Projects 

Five data sets are used to describe the characteristics of a 
project as a whole: (^) the objectives of, rationale for, and contribu- 

tions to be made by a project; (y the timelines established for complet- 
ing work within a project; (£) the organizational structure within whi i 
the work of a project is carried out; (^) the political-institutional- 
intellectual context within which a project rests; and (£) the "dynamics” 
of project operation. The first three data sets are self-explanatory. 
Context data pertain to the relationship of the project (ieing studied 
to its sister projects, to the activities of the administrative unit 
within which it rests, and to the broader political-institutional context 
within which it rests. These relationships are portrayed in the form 
of a ’’context map." 

As used in the Oregon Studies, "project dynamics" is a catch-all 
term that involves information pertaining to procedures, feelings, 
patterns of behavior, or anything else that can be used to convey a 
sense of either the "essence" of or the "reality" of working within a 
particular project. The focus of that which is reported may be project 
operations, factors influencing project operations, and/or the 
consequences of project operations. Operationally, the data pertaining 









to project dynamics involves the pooled perceptions, observations, 
hunches, and insights gained by the staff of the Oregon Studies during 
the three to five day on-site visit required for project analysis. 

No formal category sets have been developed for coding any of these 
data. All are reported in the form of narrative statements within the 
context of the case profiles. 



Data Sets Used in Describ - 
ing Project Personnel 

Three data sets are employed in describing project personnel: (^) 

the background of training and work experience of professional staff; 

(^) a description of the job or jobs held by professional staff; and 
(£) the support seirvices and resources available to staff in the perform- 
ance of their respective job roles. All of the data within these sets 
are reported in terms of questionnaire items. 



Data Sets Used in Describing Work 
Activities Associated With Job Roles 



Two data sets are employed in describing work requirements associated 
with job role: (£) the perceived requirements associated with a particular 

job held; and (_b) the emphasis given to various classes of work activities 
within the context of a particular job held. These data are also reported 
in terms of questionnaire items. 



Data Sets Used in Describing Wor k Requirements 
Associated Wiuh the Producti o n of Project Outputs 

Four data sets are employed in describing work requirements associated 
with the production of project outputs: (£) the outputs of work effort 

per se; the standards held for those outputs; (£) the operations 

required to produce specified outputs to specified standards; and (^) 
the knowledges, skills^ and sensitivities required to carry out those 
operations. These are the primary data sets reported in the profiles, 
and as such they are far more complex and extensive than the other 
data sets reported. 

In attempting to describe the outputs of projects, and the standards, 
operations, and enablers that relate to them, it was necessary to establish 
a number of category sets to handle the complexity that was found. Two 
approaches were taken to the development of these sets: (£) a conceptual- 

empirical (deductive) approach; and (jb) an empirical-conceptual (inductive) 
approach. In the fomer, category sets were developed as an extension 
of the conceptual framework that guided the Studies;^ in the latter, they 



For a description of the conceptual framework that guided the empirical 
thrust of the Oregon Studies see Schalock, H.D. and Sell, G.R. , **A Frame- 
work for the Analysis and Empirical Investigation of Educational RDD&E,** in 
Chapter 4 of Volume III of the series of volumes reporting the Oregon Studies. 



were developed in response to the data emerging from the study of ongoing 
projects.^ Operationally, however, the two approaches were complementary, 
for the conceptual-empirical approach yielded category sets that functioned 
as relatively broad, general organizers of the data, and the empirical- 
conceptual approach yielded category sets that functioned at a "close to 
the source," descriptive level. Figure 1 provides a summary of the concep- 
tually derived sets used to organize information about project outputs, 



standards , 


operations, and enablers. Figure 


2 provides 


a summary of Llie 




OUTPUTS 


STANDARDS 


OPERATIONS 


ENABLERS 




STRUCTURE 


Products 

Events 

Conditions 


Output 

Process 




Knowledge 

Skill 

Sensitivity 


FUNCTION 


Policy Setting 

Management 

Production 








CHARACTER 


Knowledge 

Technology 

Implementation 

Information 








LEVEL 


Focal 

Component 

FacilitAtiiig 




Activities 

Tasks* 

Actions 




^ 



FIG. 1. Category sets used to describe at a buoad, conceptual 
level the properties of outputs, standards, operations, and enablers. 

*0f this set, only task level descriptions were obtained. Time 
and resources did not permit an analysis of operations at the 
level of actions, and the activities set was left to be derived 
empirically. 



empirically derived category sets used to organize the same information, 
that is, statements describing work requirements in the language of persons 
working in the field. The various primary and cluster categories that 
make up these sets, as well as the procedures followed in their develop- 
ment, are described in Chapter 4 of Vol. I of the series reporting the 
Oregon Studies. The number of data statements (interviewee statements) 
classified within these various category sets include 1148 that are 



^To 3ome extent this is an over simplification, for the conceptually 
derived categories were tested empirically in the course of their 
derivation, and the empirically derived categories were always influenced 
by conceptual considerations. (See Chapter 2 in Volume I of the series 
of volumes reporting the Oregon Studies, or Volume V, for a discussion 
of the procedures followed in the development of the methodology.) 



descriptive of outp'it standards, 3722 that arc descriptive of output 
related tasks, and 2497 that are descriptive of output related enablers. 



OUTPUTS OUTPUTS 

IDENTIFIED ANALYZED STANDARDS TASKS ENABLERS 



Number of PRIMARY 
Categories Used to 

Classify Inter- 299 167 

vlewee Statements 



79 280 136 



Number of CLUSTER 
Categories Used to 

Classify Primary 45 20 

Categories 



FIG. 2, Category sets used to describe at a "close to the 
source," empirically derived level the properties of outputs, 
standards, tasks, and enablers. 



The Interdependence of Data Sets 

As indicated in the preface, each case profile was to describe not 
only the variables listed in the preceding paragraphs, but their inter- 
dependencies as well. This in turn required that a way be found to 
collect data on those Interdependencies. Accordingly, a schema was de- 
veloped which placed the full set of variables within the context of an 
interacting whole. Within this context OUTPUTS were adopted as central, 
that is, all other data sets were linked to them. Procedurally, this 
required that outputs of work effort within a project be identified, a 
set of these be selected for analysis, and for each output analyzed 
establishing the STAIJDARDS set for its production, the OPERATIONS re- 
quired for its production, the ENABLING KNOWLEDGES, SKILLS, and SENSI- 
TIVITIES needed for its production, the PERSONS Involved in its 
production, and the RELATIONSHIP of that output to the other outputs 
involved in the work of a project as a whole. It was also possible to 
link a particular output to the organizational structure of a project, 
the context within which the project rested, and even the "dynamics" 
of a project, though not so directly as in the case of variables that 
depended upon output linkage for their definition. The interaction of 
these various classes of data is illustrated schematically in Figure 3. 
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Procedures Followed in Data Collection 



Three relatively distinct procedures were employed in collecting 
tlie data reported in the profiles. Uy and large these corresponded 
with the three major classes of data collected. All data, howevery 
were collected within the context of fn "on-site" visit by a data 
collection team from the Oregon Studies. Depending upon the size and 
complexity of a project, teams consisted of from two to six people, 
and the length of the site visit extended from three to five days. 

An overview of the procedures used in collecting the various classes 
of data reported in the profiles is provided in the paragraphs that 
follow. Additional detail is provided in Chapter 4 of Volume I, and 1 
Volume V, of the series of volumes reporting the Oregon Studies. 

General Project Descriptors 



Probably the best label for the procedures used in collecting 
data on general project descriptors is that of "non-obtrusive . The 
objectives of a project, the rationale for a project, project time 
lines, organizational structures, and the like, were obtained from 
project proposals and other documents descriptive of the project. 

Also, information on the "dynamics" of the projects were gathered 
through incidental observation, the recall of casual comments made 
by project staff while being interviewed, and the "hunches" or 
"insights" gained while working with project data. Almost without 
exception these sets of data were able to be collected without in- 
trusion upon people ''s time and energy. 

The one data set used to describe the general characteristics 
of projects that was intrusive was the data set that described the con 
text within which the project rested. Some information of this kind 
was usually able to be gained from proposals and other documents, but 
in all cases project directors were interviewed when developing a 
context map. In some instances this amounted to little more than 
confirmation of information gained elsewhere, but in others it in- 
volved both the generation and piecing together of information about 
intra- and inter-institutional linkages that were simply not made 
explicit in existing materials. Generally speaking, the larger the 
project the more complex its political-institutional-intellectual 
linkages, and in some cases, for example the Children's Television 
Workshop, the development of a map to depict these linkages was a 
major undertaking. 

Personnel and Work Activity Descriptors 

All of the data that describe the personnel associated with a 
project, and all of the data that describe work activities associated 
with job roles, were collected through questionnaires. These were 
administered by members of the Oregon Studies staff, either while 
visiting the project site or through telephone. Three questionnaires 
were involved: (^) a general project questionnaire; (b) a job/ task 

inventory; and (£) a general activities questionnaire. The data 
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reported from the three questionnaires are referred to in the profiles 
as form 02, 03, and 04 data respectively# Copies of the three question- 
naires may be found in Volume V of the series of volumes reporting the 
Oregon Studies. 

Output and Work Requirement Descrintors 

All data on outputs and work related to their production were col- 
lected through interview. The interview strategy called for: (^) 

identifying', outputs associated with a project (an output index); (b) 
ordering those outputs according to their interdependencies (an output 
map); (£) selecting from the map those outputs for which work require- 
ment data were to be obtained; (^) identifying persons most directly 
responsibli* for and/or most directly involved in the production of those 
outputs; and (e) interviewing those persons in relation to the standards 
held for the output being analyzed, the tasks required to produce the 
output, and the knowledges, skills, and ensitivities needed to perform 
the required tasks. The selection of outputs to be analyzed was done by 
the data collection team, on site, after an output map had been estab- 
lished and a sense had been gained as to the outputs that were most 
critical to the project. Persons interviewed provided information 
relative to his or her own contribution to the production of a particular 
output, as well as the contributions of others (a distinction between 
self-other data was maintained throughout the project.) As familiarity 
with a project grew, adjustments were made as needed in the output map, 
the selection of outputs to be interviewed around, and the matching 
of interviewees with outputs. All interviews were tape recorded, and 
all data were reduced from the recordings by the person who did the 
interviewing. 

The reduction of the interview data involved a nultistep process: 

(a) editing tapes to Identify data statements within them, that is, 
statements pertaining to standards, tasks, and enablers; (^) the recapi- 
tulation, or '*recapping”, of data statements into a readable, gram- 
matically correct form, that is, independent clauses and/or sentences 
(care was taken not to destroy the original language of the interviewees 
in this process); (c) the transfer of the recapped statements to color- 
coded summary sheets that corresponded to the various data sets being 
used; (^) the coding of the recapped statements by a two person coding 
resolution team (during this process the coding team was free to call 
upon members of the data collection team for statement clarification, 
interpretation, context building, etc.); and (e) the storage of the 
coded data in computer files in a way that permitted the interdependencies 
within the data to be maintained. A record of all steps in the data 
collection and reduction process was maintained from the time of first 
contact with a project until all data on that project had been computer 
stored and verified. 




Tlic Trustworthiness of the Data 



Since the classes of data reported in the profiles were collected 
by various means, each must be considered separately as to Its trust- 
worthiness. Accordingly, the potential sources of error that reside 
within each data class, and the steps taken to control them, are reviewed 
in the paragraphs that follow. 

General Project Descriptors 



Four of the five data sets used to describe the general character- 
istics of projects made use of working documents. These included project 
objectives, timelines, organizational structures, and context maps. 
Typically, the document used had been prepared by project directors. 

To the extent that such documents can be accepted at face value, and 
to the extent that the Oregon Studies staff did not introduce error in 
reporting the substance of those documents, the data sets that made use 
of them were subject to few sources of error. As a consequence, no 
formal measures of trustworthiness were prepared for them. 

Judgments relative to the trustworthiness of the data reported on 
project dynamics is another matter. It will be recalled that these data 
consist of the pooled observations, hunches, "insights,” and choice tid- 
bits of information gleaned by members of the data collection team from 
a wide variety of sources. It will also be recalled that these data 
intentionally were to be subjective and impressionistic. As a means of re- 
ducing gross error all final descriptions of the dy: ^mics of projects 
were read and confirmed by all members of the data collection team that 
visited a project, buc no formal measures as to the trustworthiness of 
such data were obtained. For purposes of profile presentation, however, 
the data on project dynamics are reported. 

Personnel and Work Activity Descriptors 

Since the data sets describing personnel and work activities were 
derived through questionnaire methodology they were subject to all the 
sources of error known to operate within that methodology, for example 
the error that is introduced through the selection of questions asked, 
the possibility of multiple interpretations of those questions, and the 
lack of opportunity to determine falsification or shoddiness of response 
to the questions. The steps taken to control these sources of error were 
of two kinds: (a) reasonable care in the development and test-'ng of the 
questionnaires prior to their utilization for purposes of data collection; 
and (^) the administration of the questionnaires while the data collection 
team was on site. The first step involved a number of field trials of 
the questionnaires, and a number of revisions in them on the basis of 
those trials. The second allowed the questionnaires to be introduced 
within the context of the data collection effort as a whole, and within 
that context an opportunity to clarify troublesome questions about or 
within them. In combination, it is believed that these procedures 



Huf r iciontly reduced the typical sources of error thut er.tet tlie collec- 
tion of questionnaire data that the data reported can be viewed with a 
fair degree of confidence. 

Output and Work Requirement Descriptors 



Just as the personnel and work activity data were subject to the 
error typically associated with use of questionnaires, the output and 
work requirement data, since it was collected through intervi'ws, were 
subject to the error typically associated with interviews. Four sources 
of error have always been troublesome in this regard: (a_) the selection 

of interviewees as data sources; (b) the information elicited from inter- 
viewees about work requirements; (c) the coding of the information 
obtained from interviewees; and (d) the storage, retrieval, and analysis 
procedures used in manipulating the coded data.^ The procedures followed 
in the Oregon Studies to combat these sources of error are summarized in 
Table 1. Given the procedures followed, and the coding reliability 
obtained, it seems reasonable to view the output and work requirement 
data with a good deal of confidence. 



Profile Organization 



It will be recalled from reading tlie NOTES on the development of 
the profiles that three variations in profile format will be found in 
the present volume. These correspond to variations in the nature of the 
data collected at various points in the Studies, and represent one of 
the less fortunate consequences of the decision to emphasize methodo- 
logical development (see Preface). Although the differences i , the data 
presented in the three profile formats are not great they can be con- 
fusing to a reader when first encountered. The purpose of this section 
of the GUIDE is to introduce the reader to the general organization of 
the profiles, and to spell out how the two earlier profile formats (Formats 
1 and 2) differ from the final format (Format 3). 



3 When the profiles are being considered as a data base for cross-project 
analyses, other sources of error must be considered. Two critical 
sources are (a) the adequacy of the sample of projects drawn and (^) the 
adequacy of the sample of outputs selected for analysis within a given 
project. These are sources of error that relate to the generalizability 
of data, however, and are not of primary concern in considering the case 
profiles as descriptions of individual projects. 






TABLE 1 



Proceduras Followed in Controlling Sources of Error 
In Output-Work Requirement Data 







SOUrvCE OF ERROR 


PROCEDURES FOLLOWED TO REDUCE ERROR 



Interviewee Only staff intimately acquainted with or involved in 

Selection the production of an output were selected for inter- 

view. The relationship of the interviewee to an 
output was always confirmed by the project director, the person to be 
interviewed, and the immediate supervisor of that person. Data re- 
ported by an interviewee on the work of others in relation to an out- 
put were noted and coded separately. 

Data Generation A structured interview procedure was used to obtain 

data on the standards, tasks, and enablers associated 
with a particular output. In the interview, standards were the first 
to be identified, followed by the tasks engaged in to produce the out- 
put to those standards, followed by the knowledges, skills, and sensi- 
tivities drawn upon in carrying out the tasks identified. Stylistic 
variations in interviewing were permitted so as to acconunodate either 
interviewer or interviewee differences, but during the course of an 
interview all data sets were exhausted. (For a detailed discussion 
of interview procedures see Volume V of the series of volumes report- 
ing the Oregon Studies). 

Data Reduction A carefully established set of procedures and 

decision rules were followed in "recapping" the 
interviewee statements, and in coding the recapped statenents in 
terms of appropriate data sets. The recapped statements were first 
checked for their completeness and adequacy by the data coordinator 
upon t return of the data collection team from a project site. 

They were checked again by the coding team# Incompleteness, or 
error, or lack of clarity detected on either of these checks required 
that the recapped statements be revised until they ware acceptable 
at both quality assurance checkpoints. To insure reliable coding, 
team coder agreements were calculated. Using the recapped platements 
in three case profiles as a base for calculating coder reliability, 
and separating first and second codings by a three month period, 
coding agreements for items in each data set, with one exception, 
ranged between .69 and .96. Reliability in coding task statements 
was .60. Detailed coder reliability data are reported in Chapter 4 
of Volume I of the series of volumes reporting the Oregon Studies# 

Data Storage As soon as the recapped statements had been coded 

and Retrieval for a particular project the codes were forwarded 

to the data coordinator for a check of their com- 
pleteness, and then forwarded to the coordinator of data storage and 
retrieval for transfer into computer storap:e. After storage, repeated 
checks were run to insure that the initial computer entries were 
correct, and the computer center manipulations over time had not 
destroyed or reordered the data as it was originally stored. 
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Profile Format 1 



Ten profiles in the volume meet the most advanced format require- 
ments. 4 These are profiles 1, 2, 6 and 7 in Part One of the volume; 
profi'es 9, 10, and 11 in Part Two; and profiles 16, 17, and 18 in Part 
Three. As a set these profiles reflect the most .idvanced form of the 
data collection methodology, were the last to be prepared, and appear 
as the first profiles to be read in any of the three parts to the 
volume, as well as the first to be read in the llvaluation section of 
Part One. Also, all are organized into six chapters: an Overview; 

a Description of the Parameters of the Project; a Summary of Data; 
Supplementary Data; Project Dynamics; and Implications for Training. 

Each profile also contains an Appendix that houses the "recapped" data 
statements from which the output-work requirement data summaries have been 
prepared. 

An overview of the contents of each chapter in the format 3 profiles 
follows. It will be seen from these overview statements that the three 
classes of data collected on a project are collapsed and/or integrated 
for purposes of their presentation within profiles. 

CHAPTER I: OVERVIEW. This chapter provides the first view of a 

project as more than a title. It provides an orientation to the nature 
of the project, its goals, and its reasons for being, and serves as the 
framework into which the balance of the profile data are fit. Structurally, 
the overview chapter consists of the following parts : 

(£) Synopsis of the Project 

(^) Objectives, Rationale, and Significance of the Project 

(O Context in Which the Project Operates 

Chapter I -.s generally not more than 6 pages in length, and it is 
designed as an "abstract" so that readers may determine whether they 
wish to read the profile as a whole. 

CHAPTER II: PARAMETERS OF THE PROJECT. Chapter II emphasizes, 

and makes quickly available, a first set of "hard" data about a project. 
Standard sections include: 

(£[) Staff structure; 

(^) Project roster; 

(O Index of outputs; 

(^) Output map. 

Staff structure data involves a description of the organizational structure 
adopted by a project, and how staff members are distributed within that 
structure; project roster data involves a description of the roles 
played and/or functions performed by personnel within the project; an 
output index is an annotated listing of the outcomes of work effort 
that project staff identify as critical to the success of the project; 



4 Ea profile Is identified as to its format number on the back of 
the profile title page. 
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and an output map is a schematic portrayal of the interdependencies 
between project outputs. More is said about output maps later in the 
GUIDE. 

CHAPTER III: SUMMARY OF DATA. In terras of the data sets described 

previously, this chapter would be more accurately titled '^Summary of 
Work Requirements for Output Production.” Three data sets are summarized 
in the Chapter; (a) the standards held for the production of an output; 
ih) the tasks engaged in to produce an output to the standards set for 
it; and (O the knowledges, skills, and sensitivities required to perform 
those tasks. Each of these data sets is displayed in standard tables 
as frequencies of category citations. The narrative text of the chapter 
deals principally with the data displayed in the tables, and the inter- 
relationships of those data. 



CHAPTER IV: SUPPLEMENTARY DATA. The chapter on supplementary data 

varies to some extent as to the specific data it contains. In general, 
however, the following data sets are reported: 





(a) 


Kinds of outputs generated at varying stages of project 
completion ; 




(b) 


The distribution of outputs by their alternative classifi- 
cations, i.e., structure, function, character and level; 




(£) 


Summaries of staff backgrounds; 




(d) 


Individual job descriptions; 




(e) 


Interviewee responses to questionnaire items relating to 
position requirements, support resources, and project 
management ; 




(f) 


Interviewee responses to questionnaire items citing 
emphases given to various classes of work activities; 




(£) 


The funding base of the project. 


Tables of the data are provided when they serve to provide a focus to 
the discussion- Meaningful relationships with data reported in other 
chapters are also pointed out. 


CHAPTER V 
structured of 


: PROJECT DYNAMICS. This chapter, by design, is the least 

the profile chapters. The purpose of the chapter is to 



round out the profile by reporting ”impressionistic” observations about 
the project. The ”data base” for the Chapter was the hunches, observa- 
tions, insights, etc. gained by the data collection team during their 
three to five day stay at the site of the project. Thufif? Imprewslone 
are reported in whatever sequence, form, and substance the profile writer 
considered best in calling out the significant and unique features of 
project operation. The freedom of the dynamics chapter to vary in focus 
and content was considered essential to extending the meaning of the 
data collected. It was also seen as essential to methodological develop- 
ment, for it served as the vehicle by which new data thrusts were identi- 
fied for inclusion in the methodology. 

r 



The substantive focus of the comments included in most project 
dynamics chapters includes some subset of observations with respect to 
staffing patterns, project management structures and procedures, manage- 
ment ^’styles,’* project related commitments, substantive issues that 
arise within projects, affective issues, and agency interrelationships. 

The discussion of such observations is linked, when appropriate, to 
^*hard” data. Tlie tenor of the discussion is intended to be non-judgmental 
and instructive. 

CHAPTER VI: IMPLICATIONS FOR TRAINING. In this chapter the know- 

ledge gained about a project is assessed with respect to its implications 
for training. In this assessment the data reported in the profiles 
generally are treated very briefly, for it Is assumed that the reader 
can draw his own conclusions from his reading. Instead, attention 
is directed to comments or recommendations made about training by project 
personnel, or which are implied by the nature of the data collected. The 
discussion frequently focuses on training needs mentioned by project staff 
in relation to problems or difficulties in the project. To this extent, 
the discussion tends to highlight areas of competence in which prepara- 
tion was weak. 

PROFILE APPENDIX. The last chapter in each profile is followed by 
an appendix that contains the "raw*' data that is the basis for the coded 
data reported in Chapter III. The raw data consists of the paraphrased 
or "recapped" statements of interviewees that describe the standards, 
tasks, and enablers associated with the generation of outputs. Category 
code numbers are included with each statement to facilitate their loca- 
tion in the various tables presented in Chapter III. The importance of 
this appendix extends beyond its function as an aid to the reader, for 
it represents what is presumed to be one of the most meaningful forms in 
which the data collected in the Oregon Studies can be presented for 
purposes of training. Furthermore, the profile appendix is the only place 
where the raw data on standards, tasks, and enablers appear. 

Profile Format 2 

Six profiles in the volume were prepared according to the format 
that proceeded in time the format just described. These are profiles 3, 
A^and 8 in Part One of the volume; 12 and 13 in Part Two; and 19 in 
Part Three. 

The main differences between formats 2 and 3 lie in the language 
used to describe project outputs. In format 2 the language of output 
structure, function, character, and level was not in use, and the distinc- 
tion between products, events, and conditions had not as yet emerged. In 
their place was a language of production and management "products," 
where products served as a loosely defined term to cover what subsequently 
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was recognized as products, events, and conditions. 

These differences are reflected in the content of Chapters II and 
III of the format 2 profiles. In all other respects both the content 
and organization of format 2 profiles are consistent with those reported 
in format 3. 

Profile Format .1 

Four profiles reported in the volume were prepared according to the 
first profile format developed. These are profiles 5 in Part One of the 
volume; 14 and 15 in Part Two; and 20 in Part Tliree. Since the varia- 
tion between formats 1 and 3 is considerable, differences will be traced 
chapter by chapter. 

CHAPTER I. Same as in format 3. 

CHAPTER II. In place of an output index and an output map there is 
(a^) an index of production responsibilities, (^) a production responsi- 
bility tree, (c) an index of management responsibilities, and (jd) a 
management network. These correspond to the output index and output map 
of format 3, £nd for purposes of data analysis were so treated, i.e., 
they were recoded using the data sets reported in format 3 profiles. In 
format 1, production responsibilities are treated much as products are 
treated in format 3, and management responsibilities are treated much 
as events and conditians ara treated in format 3. The distinction 
management and production responsibilities, however, are carried into 
Chapters III and IV of format 1, causing two chapters in the profile to 
be devoted to work requirement data (such data are consolidated in 
Chapter III in formac 3). As is the case in format 2 profiles, the 
language of output index and map, and the language of output structure, 
function, character; and level does not exist. 

CHAPTER III. Entitled DETAILS ON EACH PRODUCTION RESPONSIBILITY, 
this chapter presents the data on standards, tasks, and enablers only 
for products pertinent to the contractual obligations of the project. 

The chapter also contains the recapped interviewee statements (in format 
3 profiles they appear as an Appendix), as well as the category frequency 
data that are based upon these statements. 

CHAPTER IV. Entitled DETAILS ON EACH MANAGEMENT RESPONSIBILITY, 
the chapter simply repeats the format of Chapter III. 

CHAPTER V. Equivalent to Chapter IV in format 3. 

CHAPTER VI. Equivalent to Chapter V in format 3. 



Subsequent to the preparation of format 2 profiles; production and 
management “products” were reclassified into products, events, and 
conditions. Two purposes were served by this reclassification: (^) 

it eased the strain of what had come to be recognized as a forced 
classification; and (^) it enabled the data reported in these profiles 
to be used in cross-project analyses. The recoded data are reported 
in supplementary tables that accompany each format 2 profile. 







CHAPTER VII. Equivalent to Chapter VI in format 3.^ 



Notes on Reading Output Maps 



The output map found in each of thc: profiles contains a wealth of 
information about the outputs of the project under investigation. In 
order to extract all the information that a map contains it is essential 
that the rules guiding the construction of a map be understood. 



The Purpose of the Map 

The purpose of the output map is to present as simply and as 
clearly as possible the interrelationships that exist between the various 
outputs of a project. The desired effect of reading an output map is 
a ^picture'* of the project being discussed in terms of tlie dependency 
relationships among the outputs the project seeks to achieve. 



The Elements in a Map 

Figure 4 contains an illustrative output map. A number of elements 
can be identified within it: solid line boxes; labels; code symbols; 

horizontal lines; vertical lines; brackets; dotted lines; dotted line 
boxes; and vertical (long and short dash) lines. Each of these elements 
contributes to the total information contained in a map. The following 
paragraphs identify the information presented by each element. 

Solid line boxes 



I Each solid line box represents a specific output 

that the project is seeking to achieve. If the 
box stands alone (is not connected to any other 
box by a line) one of two conditions exists: 

(^) the output is considered to have value, but is 
not related to any other output, or (^ the output 
index did not contain output identifications that 
allowed other outputs to be linked to it. 

Labels 



Within each box there is a label which is Lhe 
descriptor of the output represented. The labels 
found in the box are the same ones used to describe 
a particular output throughout the profile. 



Final 

Report 



Profile 14 was a transition profile, and is peculiar in that it 
incorporates the language of the profile 1 format but the organization 
of profile 2 and 3 formats. 
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Evaluation Reporta 
ALERT (Producta) 

P-19 



Coded symbol 



Each box contains, along with the label, a coded 
symbol. Each symbol is composed of a letter which 
identifies the structure of the output (P-product, 
E-event, C-condition) , and a 2-digit numeral which 
identifies the output sequentially with respect to 
the other outputs in the same profile. Code 
numbers are the same throughout the profile. 

Horizontal lines 



Horizontal lines between boxes indicate that the 
outputs so connected have side-effect relationships, 
that is, th'^ production of one influences the other, 
and vice versa. 



P-01 Final 
Report 



Vertical lines 




Vertical lines between boxes indicate that the upper 
output is dependent on the lower. Until the lower 
output is completed the upper one cannot be completed. 
In the total map, boxes connected by vertical lines 
are hierarchically arranged, those at the top of the 
map being dependent upon all those below. 



Boxes enclosed by brackets represent outputs which 
are influenced by, but not dependent on the output 
represented by the box linked horizontally to the 
bracket(s) . 



Dotted lines 

Dotted lines connecting two brackets are used to 
indicate outputs that are influenced by another 
output when the outputs encompassed by a bracket 
area are large in number, or when influence is 
carried across more than one page of the map. 




Dotted line boxes 



I 

I 

I 



I 

I Dotted line boxes represent those outputs which 
- are either generated outside the project, but 
influence it, or are outputs generated by the 
project as a function of other outputs but have 
not been indexed by project staff. 



Vertical (long and short dash) lines 

^ When more than one page is needed to display an 

( output map, vertical lines (long and short dash) are 
used on the right of the first page and the left of 
I the second page to indicate the point at which the 

I two pages coincide. Q 
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Under unusual circumstances some outputs may appear more than 
once in a map. This results when they are related to other outputs 
in differ int ways, i.e., dependent on one set but influenced by another. 
Wlien it is impossible to display both the relationships by one place- 
ment, outputs are repeated. 

Once the purpose of an output map is understood and the various 
elements within it are defined, the reader should be able to extract 
a great deal of information from a careful analysis of a map. It 
should be made clear, however, that an output map does not attempu 
to display time relationships as do other process charts such as PERT. 
An output map focuses on the dependency relationships existing between 
outputs, independent of the factor of time. 
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GLOSSARY OF COMMON PROFILE TERMS 



This glossary contains definitions of terms used frequently in the pro- 
files. Asterisks identify terras that were used in the early forms of 
the profiles. These terras, no longer in use, are identified with a 
single asterisk to indicate their appearance in profile format 1 and a 
double asterisk to indicate their appearance in profile format 2. When 
terms are used in a definition that are themselves defined in the glossary, 
they appear in capital letters. 



ADOPTION. A circumstance in which KNOWLEDGE, INFORliATION , and/or 
TECHNOLOGY is utilized. 

CHARACTER OF OUTPUT. See Output Character . 

COMPONENT OUTPUT. An outcome of work effort that constitutes an element 
of, or an approximation to, a FOCAL OUTPUT. 

CONDITION. An outcome of work effort that creates a desired circumstance 
expected to endure over the life of a project, or as a result 
of it. 

CONTEXT. See Project Context . 

DEVELOPMENT. A problem-solving strategy designed to produce reliable 
technology, that is, procedures, materials, hardware, and 
organizational frameworks that have a known degree of success 
in bringing about a particular outcome or in performing a 
defined operation; also used to designate the focus of 
projects (see Project Focus) and the focus of outputs (see 
Output Focus ) . 

DIFFUSION. A problem-solving strategy designed to bring about the 

implementation of generalizable knowledge, a reliable tech- 
nology, or trustworthy information (as used here diffusion 
incorporates both the concepts of DISSEMINATION and ADOPTION); 
also used to designate the focus of projects (see Project Focus ) 
and the focus of outputs (see Output Focus ). 

DISSEMINATION. A circumstance in which KNOWLEDGE, INFORMATION, and/or 
TECHNOLOGY is distributed to a targeted population. 

EDUCATIONAL RDD&E. A coordinated set of problem-solving strategies 
designed to produce outputs that can be judged as to their 
quality and their contribution to the solution of educational 
problems. 

ENABLER. KNOWLEDGES, SKILLS, and SENSITIVITIES needed to produce a 
particular output. 



KNVIRONMI-NTAi. MA^NAGLMLiNT RKSPONSIurLITY / Responsibilities which, when 
carried out, result in outcomes that enhance or facilitate tlie 
environment in which a project operates, (Subsequently, only 
of these responsibilities were analyzed.) 

EVALUATION, A problem-solving strategy designed to produce trustworthy 
information regarding a phenomenon which occurs in a context 
or enviromaent over which the user expects to exercise 
influence or about which he expects to make decisions; also 
used to designate the focus of projects (see Project Focus ) 
and the focus of outputs ^,see O utput Focus ), 

EVENT. An outcome of work effort that results in the occurrence of an 
observable t rans£ic t ion or set of behaviors . 

FACILITATING OUTPUT, An outcome of work effort that supports the 
generation of FOCAL or COMPONENT OUTPUTS, but is not in 
itself a part of such outputs, 

FOCAL OUTPUT. An outcome of work effort expected by contractual 
obligation to emerge from a proje 

FOCUS. See Project Focus and Output Focus . 

FUNCTION. See Output Function , 

IMPLEMEN'^^TION . A classification given an output of DIFFUSION, i.e., 
an instance of the ADOPTION and UTILIZATION of KNOWLEDGE, 
INFORMATION, and/or TEClfNOLOGY ; the objective of DIFFUSION. 

INFORMATION. A classification given an output of EVALUATION, i.e., an 
instance of reliable information about a given plienoraenon 
within a context over which a user expects to exercise 
influence or about which he expects to make decisions; the 
objective of EVALUATION. 

KNOWLEDGE (AS ENABLER), A classification given an ENABLER that 

identifies it as a fact, principle, or generalization, and tuat 
can stand the test of empirical verification; also, any 
circumstance that can be shown to exist, 

KNOWLEDGE (AS OUTPUT). A classification given an output of RESEARCH, 
i.e,^ an instance of established fact, principle, etc. that 
is generalizable and that can stand the test of empirical 
verification; the objective of RESEARCH, 

LEVEL OF OUTPUT. See Output Level , 

MANAGEMENT FUNCTION, A classification given an output that orchestrates 
the resources (time , personnel , materials , space , information) 
available to a project for the realization of the outcomes 
expected from it; also a report of that orchestration . 
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MANAGIMKNT NETWORK (also MANAGEMENT RESPONSIBILITIES NETWORK).* A 

hierarchical ordering that graphically illustrates the functional 
relationships between MANAGEMENT PRODUCTS and RESPONSIBILITIES 
within a project. (Subsequently incorporated within output 
mapj . ) 

MANAGEMENT PRODUCT. A classification given a product serving a 

MANAGEMENT RESPONSIBILITY. (Subsequently identified as a 
product serving a MANAGEMENT FUNCTION.) 

MANAGEMENT RESPONSIBILITY.* See Environmental Management Responsibility 
and Production Management Responsibriity . 

MANAGEMENT RESPONSIBILITY INDEX.* A listing of the MANAGEMENT RESPONSI- 
BILITIES within a project. (Subsequently incorporated within 
the OUTPUT INDEX.) 

OUTPUT. An identifiable outcome of targeted work actlv^ity that con- 
tributes to the realization of project goals. 

OUTPUT CHARACTER. The attributes of an output that mark it as an instance 
of KNOWLEDGE, TECHNOLOGY, IMPLEMENTATION, or INFORMATION. 

OUTPUT FOCUS. The attributes of a FOCAL OUTPUT that mark it as an output 
of RESEARCH, DEVELOPMENT, DIFFUSION, or EVALUATION. (In 
Format 1 and 2 profiles, all outputs are classified in terms 
of an RDD or E focus.) 

OUTPUT FUNCTION. The attributes of an output that mark it as serving 
a POLICY, MANAGEMENT, or PRODUCTION FUNCTION. 

OUTPUT INDEX. An annotated listing of the outputs of a project. 

OUTPUT LEVEL. The attributes of an output that identify its relationship 
to project goals as FOCAL, COMPONENT, or FACILITATING. 

OUTPUT MAP. A graphic portrayal of the functional interdependencies 
among the outputs of a project. 

OUTPUT STANDARD. A criterion applied to, or level of excellence expected 
of, an output; a criterion by which the adequacy of an output 
is Judged. 

POLICY FUNCTION. A classification given an output that establishes 
standards or guidelines for a project. 

PROCESS/OPERATIONS STANDARDS. A criterion applied to, or level of 

excellence expected of, the processes/operations engaged in 
in producing an output; a criterion by which the adequacy 
of processes/operations are judged. 

PRODUCT. A tangible or **hard** outcome of work effort, concrete in 
form, and transportable at a given point in time. 



O 

ERIC 



PRODUCTION FUNCTION. A classification given an output that is a part 
of the total fabrication effort of a project. 

PRODUCTION MANAGEMENT RESPONSIBILITY."^ Responsibilities which, when 
carried out, result in outcomes that enhance or facilitate 
the generation of products for which the project is responsible. 
(Subsequently, only the outputs of these responsibilities were 
analyzed, ) 

PRODUCT TREE or PRODUCTION RESPONSIBILITY TREE."^ A graphic portrayal 
of the functional interdependencies among the products of a 
project (equivalent to an OUTPUT MAP, except it contains 
only PRODUCTS). 

PROJECT. A formally recognized, funded and directed effort aimed at 

achieving one or more specified ends that have their definition 
in educational RESEARCH, DEVELOPMENT, DIFFUSION, and EVALUATION. 

PROJECT COMPLEXITY. A project dimension defined in terms of level of 
funding and duration. 

PROJECT CONTEXT. A project dimension defined in terms of institutional 
setting, e.g., schools, colleges and universities, publicly 
supported laboratories and R&D centers. 

PROJECT FOCUS. A project dimension defined in terms oi primary emphasis 
of work effort, i.e., RESEARCH, DEVELOPMENT, DIFFUSION, and 
EVALUATION. 

RESEARCH. A problem-solving strategy designed to produce reliable 
KNOVTLEDGE , that is, facts, principles, theories, and laws 
that are generalizable and that can stand the test of 
empirical verification; also used to designate the focus of 
projects (see Project Focus ) and the focus of outputs (see 
Output Focu s) . 

SENSITIVITY. A classification given an ENABLER that identifies it as 
an increment of awareness about an environment or factors 
operating in or upon an environment; also, attitudes and 
personality characteristics. 

SKILL. A classification given an ENABLER that identifies it as an 
ability, proficiency or expertness in the exercise of an 
art , craf t , or science . 

STANDARD. See Output Standard and Process/Operations Standard . 

STRUCTURE OF ENABLERS. A classification given ENABLERS that identifies 
them as KNOWLEDGES, SKILLS, or SENSITIVITIES. 

STRUCTURE OF OUTPUTS. A classification given OUTPUTS that identifies 
them as PRODUCTS, EVENTS, or CONDITIONS. 
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STRUCTURE OF STANDARDS. A classification given STANDARDS that identifies 
them as OUTPUT STANDARDS or PROCESS/OPERATIONS STANDARDS. 

TASK.. A unit of work performed in producing a specified OUTPUT to a 
specified STANDARD. 

TECHNOLOGY. A classification given an output of DEVELOPMENT , i.e., an 
instance of a plan, procedure or product that when applied 
can bring about a desired end with a known degree of reliability 
the objective of DEVELOPMENT. 

TREE. See P roduct Tree . 

UTILIZATION. A circumstance in which KNOWLEDGE, INFORMATION, and/or 

TECHNOLOGY is employed in accomplishing a goal or end state. 




O 



xxxix 



CASE PROFILE NO. 9 



Written by 
Clark A. Smith 



PROJECT TITLE: Tri-University Project on Behavioral Objectives in 

English, Grades 9-12 

(BOE Project) 



AN EDUCATIONAL DEVELOPMENT PROJECT CONCERNED WITH: Preparing and field 

testing a catalog of representative behavioral objectives in English, 
grades 9-13, together with suggested and exemplary procedures for eval- 
uation, which will aid curriculum workers and teachers of English through- 
out the country in elaborating procedures for achieving desirable out- 
comes in English instruction. 



A’PROJECT OF: University of Illinois (in cooperation with 

Indiana University and Purdue University) 
Department of English, 109 English Building 
Urbana, Illinois 61801 





This profile has been prepared according to 
PROFILE FORMAT No, 3 

Three profile formats are represented in this volume. 
The reader should refer to this number when making 
use of the reader's GUIDE to the profiles. 
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Chapter I: Overview 



The overview presents a brief synopsis of the Behavioral Objectives 
Exchange (BOE) Project. This is elaborated by a discussion of the objec- 
tives, rationale, and significance of the project and the context in which 
it operates. 



Synopsis of the Project 

Title: Tri-University Project on Behavioral Objectives in 

English, Grades 9-12. 

Responsible Institution: University of Illinois (in cooperation with 

Indiana University and Purdue University). 

Funding Source: Bureau of Research, 

U.S. Office of Education 

Funding Duration: June 15, 1969 to August 15, 1971. (26 months) 

Observation Date: May 1971. 

Present Stage of Development: Final-Stage. 

RDD&E Focus of Project: Educational development. 

Expected Outcome: A catalog of representative performance objectives 



in English, grades 9-12. 



Level of Funding and Duration: Medium, (level A of 7 levels) 



Agency Setting: University. 



Staff Summary (current) 



Professional 



Support 



Total Full Time Equivalency 
(in man years): 



2.31 



4 



Number of Personnel Assigned: 



7 



3 



Professional Specialties of Staff 
(interviewees only): 



English, English education, 
mass media, educational psy- 
chology , journalism. 
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Objectives, Ratlonalej and Significance of the Project 

The primary goal of this project is to prepare and make available 
to the profession, a field tested catalog; of representative performance 
objectives in English for Grades 9-12 ♦ Several additional purposes are 
served by the underlying philosophy of the project and the procedures 
set for carrying the project out. These include: 

1. Giving definition to a learner ^s mastery of the content 

of English, while simultaneously giving definition to that 
content itself. 

2. Legitimizing the efforts of specialists in English focusing 
on the learner as a doer, through the inclusion of repre- 
sentatives from the behavioral sciences in project activities. 

3. Identifying those outcome dimensions judged to be valid and 
important but for which immediate behaviors are not apparent. 

In so doing, maintaining a perspective on desirable outcomes 
and insights into their achievement. 

4. Avoiding the risk of superficiality in behavioral state- 
ments by including a broad range of points of view held 
by persons participating in project activities. 

A significant contribution intended by the catalog is the expli- 
cation of a broad range of representative objectives which high school 
English teachers will generally accept as indicators of the achievement 
of various goals by learners. In effect, such explication gives a 
common frame of reference around which to examine the meaning of 
various abstract terms (e.g., appreciation) used in specifying broad 
classes of desirable outcomes. 

The following is from the abstract in the project proposed: 

Representatives of the University of Illinois, Indiana University, 
and Purdue University will work for one year with approximately 
27 consultants in the preparation of a catalog of representative 
behavioral objectives for English in grades 9-12. (There will be 
two) will serve as major consultants; the others will represent 
selected ES-70 schools and various specialties in English and 
closely related fields. Before the first conference of consul- 
tants, the directors will review general aims for English teaching, 
attempt a categorization of them and prepare a sample set of 
write-ups of behavioral objectives. The first conference of 
consultants will draft statements of objectives, including 
evaluative procedures, in each category. After the conference, 
the directors will revise and edit the results and mail the draft 
to the consultants. At a second conference, after hearing com- 
ments from (the two major consultants), the consultants will 
revise and strengthen the draft. Then the directors will com- 
plete A Preliminary Catalog of Representative Behavioral 
Objectives in English , Grades 9-12 . 
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During the second year the directors will field test 
the Preliminary Catalog in eight ES-70 schools and 16 other 
selected schools throughout the country. The results of the 
testing in various control and experimental situations, plus 
the reactions of a small group of carefully selected outside 
readers, will guide the directors in revising the Preliminary 
Catalog . The final document, to be made available to the 
profession, will be A Catalog of Representative Behavioral 
Objectives in English , Grades 9^12 . 

Context in Which the Project Operates 

The context in which the Behavioral Objectives in English (BOE) 
Project operates is illustrated in Figure 1. Contractually the rela- 
tionship is between the U.S. Office of Education (USOE) as the sole 
sponsor and the University of Illinois as the contracting institution. 
Operationally, the project is directed by two faculty members from the 
English departments of each of the three cooperating institutions, plus 
an educational psychologist from Purdue University. 

Relationship to other agencies. Most of the Project 
Directors hold several positions of administrative responsibility, in 
addition to teaching responsibilities, within their respective insti- 
tutions. In each instance within the cooperating institutions the 
English Departments (and the Purdue Education Department) agreed to 
cooperate with the project by making available .33 FTE-^ release time 
for each of the Directors. These departments, as well as any con- 
cerned research units within the Universities, reviewed the proposal 
and indicated no objection in principle to its implementation. 

In addition to the Directors, consultants from the English disci- 
pline as well as the behavioral sciences were utilized in the primary 
generation activity. The list of consultants included public school 
personnel from eight ES-70 schools 2 (one of these a parochial school) and 
University faculty from various schools, departments, or divisions 
of English, education, linguistics, speech and theatre, and humanities. 
Two major consultants with expertise in the behavioral sciences, specif- 
ically dealing with the explication of behavioral objectives, were em- 
ployed as over-all consultants to the effort. 

Field test sites selected consisted of eight ES-70 schools across 
the nation, ranging from the Northwest to Texas and to the Eastern Coast, 
eight non-ES-70 schools in close proximity to the ES-70 schools, and 
eight schools near the project’s cooperating institutions. In effect, 
this permitted the testing of the catalog in schools having (^) formal 
commitment to innovation and change, (]b) those not as formally committed, 
but geographically dispersed as to permit efficient visitation logistics, 
and (O those not as formally committed but close at hand to facilitate 
planning and continuous monitoring considerations. 



^Full Time Equivalency 
2 

A national consortium of public schools committed to systems design 
principles in the development of educational systems. 
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FIG. 1. Contextual map 



Time lines . Project time lines were explicitly stated in the proposal 
and are summarized in Figure 2. The project remained essentially on schedule 
through the first year and to the point of observation in the second. One 
feature of the plan is the gap in scheduled activity immediately preceding 
each of the two work conferences. This made available to project staff time 
to take action on any unanticipated problems and insure proper preparation 
for the two critical events of the project. Differences of opinion which 
arose will be discussed in Chapter V. It is important to note, however, 
that prior planning left the project staff with a total of six to eight 
weeks of unscheduled time to be filled as the situation required. 

Physical/environmental setting . One of the more unique characteristics 
of this project is the composition of its staff and directorship. The 
entire staff consists of the seven directors, plus some secretarial help 
in three separate locations. In one respect such an arrangement has the 
advantage of centralizing all project activities and responsibilities with 
the leadership. On the other hand, the nearly equal distribution of mem- 
bers of the directorship over a geographic area ranging in distance from 
approximately 1;^0 to 150 mil^s jcreated some difficulties, particularly 
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with communication. Two mechanisms were employed to counter the effects 
of this distribution. The first was the use of the Wide Area Telephone 
System available to each of the directors through their institutions. 
Secondly, such telephone communications permitted early identification 
of the need for a collective gathering to handle project concerns or busi- 
ness. The location of these meetings was Crawf ordsville , Indiana, thus 
reducing travelling distances to a range of 25 to approximately 80 miles. 

Since the time for site visitation in the study of this project 
coincided with a time all Directors were in the University of Illinois 
area, the physical setting of the project can only be described as it 
pertains to the University of Illinois. Project offices are on the first 
floor of an older multifloor building located less than one block from the 
mall of the main campus. The building fronts on a heavily shaded, rela- 
tively narrow street and is of a size that permits small offices on each 
side of the building, running three or four small offices toward the rear, 
with a narrow hallway down the center. Essentially three office spaces 
are occupied by project personnel, two for the Illinois Directors and a 
full time secretary, and one which serves as a conference room. The im- 
pression of limited space in these areas is exaggerated by the inclusion 
of an extensive project library of references and materials within them. 
The two Project Directors do, however, maintain their own faculty offices 
elsewhere on the campus. 
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Chapter II: Parameters of the Project 



Presentation of the parameters of the BOE Tri-Univ3rsity Project 
is focused on two dimensions, (^) the project structure in terms rf the 
staffing pattern employed, and the roles and functions served, and (^) the out- 
puts generated by the project and its personnel. Interpretive discus- 
s ion , where applicable, is presented in subsequent chapters . 



Project Structure 

Staff structure . Immediate reference is made to Figure 1 in Chapter. 
I. The project staff structure may be extrapolated simply from that figure 
as in Figure 3, which is an illustration of the organizational 
structure of the BOE Project. 



Indiana University 

2 Project Directors 
(English Dept . ) 



7-man Directorate 

Univ. of Illinois 

2 Project Directors 
(English Dept . ) 

4 Inhouse consul- 
tants 



Purdue University 

3 Project Directors 
(2-Engl . , 1-Educ . ) 



2 Inhouse consul- 
tants 



2 Major Consultants 




FIG. 3. Project organizational structure. 



Operationally, responsibility for overall management and production 
activities resides within a seven man directorate, with guidance provided 
by two special consultants having the immediate ear of that directorate. 
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The primary production efforts in generation of the BOE catalog are as- 
signed to a group of working consultants. Liaison personnel from the 24 
field test sites relate to project activities just prior to execution of 
the field test of the preliminary catalog. They .ire primarily responsible 
for (^) facilitating the distribution of the catalog, (Jb) introducing its 
use to local faculty members, (c^) the processing of questions and problems 
encountered in implementation, and (^) facilitating the completion and re- 
turn of evaluation instruments. The outside readers have responsibility 
for the review and critique of various aspects of the preliminary catalog, 
with principal contribution made at the prototype stage (year one). Tri- 
University consultants. Colleagues of the Directors, are used for imme- 
diate and online consultation by each Director as he pursues the catalog 
section for which he has major responsibility. These consultants also act 
as participants in the working conferences. 

Project roster . Table 1 elaborates the staff structure of the BOE 
Project by explicating the specific job titles of personnel and providing 
an estimate of FTE commitment of each to the project. 



TABLE 1 

Project Roster of BOE Staff by Job Titles 



Directorate 



Time commitment 



7 Project directors 

Consultants 

2 Major consultants 
16 English discipline 
consultants 

9 Public school consultants 
6 Tri-University consultants 
5 Outside readers 

Liaison personnel 

24 School representatives 
from field test sites 



each at .33 FTE 



each to react as required 

each at 5 conference days 
each at 5 conference days 
each at approximately 5 days 
each to react as required 



each at 2 conference days 
plus ongoing responsibility 



O 
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Outputs Generated 

Each of the outputs generated by any collective action of a group 
of people focusing on a common production effort has been conceptualized 
as falling into one of three classes: (^) tangible products which exist 

concretely at points in time; (Jb) events or processes which facilitate or 
are requisite to achieving the project objective; and (£) conditions or 
states of being which contribute to achieving project objectives. In 
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each of these cases the output may also either facilitate, be a component 
of, or even be the focus of project objectives in and of themselves. In 
addition, each has a functional relationship to the project relative to 
production, management, or policy setting and can be classified as to 
Character, i.e. , knowledge, technology, implementation, or information. 

Index of outputs . To facilitate presentation of project outputs in a 
manner that communicates their relationships within the project, two 
modes are used: (^) output listed by level (focal, component, facilitating), 

and annotated in accordance with their structure (product, event, condition) 
and Function (policy setting, management, production), and (b^ outputs 
mapped according to their dependencies. The code number preceding each 
annotation consists of two parts, a letter and a two-digit number. The 
letter simply indicates the structure of the output, i.e., P^product, 

E=event, and C=condition, while che number is used for storage identifi- 
cation purposes. Subsequent references to, or listings of, outputs in this 
profile are in accordance with these codes. 



Focal Outputs : 

Products seifving a project management function: 

P-02 Commercially Published Edition of Performance Objectives 
in English, Grades 9-12 . An edition of the catalog, the 
publication of which has been negotiated with a known 
and established English text publishing firm for the 
purposes of broad dissemination and availability to 
school personnel. 

P-03 Final Report . A report by project management which 

satisries the obligation of the contract, declares the 
project completed, and declares that the conditions under 
which it was funded have been met. 

Product serving a project production function: 

P-01 Final Catalog of Performance Objectives in English . 

Grades 9-12 . A catalog of objectives for high 
school English systematically derived from educa- 
tional and discipline goals with implied and explicit 
measures of their achievement. 

Component Outputs : 

Products serving prrject production functions: 

P-09 Preliminary Catalog . A prototype edition of the catalog 
of performance objectives in English to be used as the 
field test version. 

P-10 Refined (conference 2) Approximation to Catalog . A col- 
lection of performance objectives created and/or refined 
and ordered by working conference consultants using pre- 
viously defined objectives as beginning points. 
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P-13 First Draft of Objectives for Chapters to Catalog . A 

collection of performance objectives specific to the areas 
of concern In each chapter of the catalog. The organization 
of each chapter according to the outputs of an initial work- 
ing conference nas been the responsibility of one or more 
directors . 

P-18 First Draft of Introduction to Catalog . An initial draft 

of an introduction to the catalog of performance objectives 
created by the initiating project investigator. It outlines 
the intents, purposes, and uses of the catalog and sum- 
marizes earlier dissenting issues arising on the project. 

P-22 Initial Statement of Goal Areas for English . A first 
approximation o5 goal statements for education and 
the discipline of English created collectively by 
the Project Directors with the guidance of colleagues 
and maj or consultants • 

Facilitating Outputs ; 

Product serving a project policy function: 

P-14 Journal Report (Project Officer) . A paper prepared by the 
sponsoring agency as an inclusion in an English discipline 
professional journal. It was designed to set forth a 
sponsor position with respect to the improvement of edu- 
cation and particularly with respect to instruction in 
English. 

Products serving project management function? 

P-05 User Questionnaires . A set of questionnaires developed by 

the project staff, designed to obtain information from teach^ 
ers and students regarding their use of the preliminary 
catalog during the field test. 

P-07 Interview Questionnaire (Guide to Interviewers) . A 

schedule of the information to be collected by project 
staff during site visitations conducted toward the end of 
the field test period. This schedule was prepared by project 
staff prior to initiating the field test year of the project. 

P-11 Modified Field T^st Plan . A revised plan for field testing 
the catalog based upon recotomendations coming from consul- 
tants during the first nine months of project operation. 

P-19 Agenda for Initial Conference . An outline of the general 
activities to be undertaken by participants in the 
initial working conference. 
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P-23 Initial Field Test Plan . An initial field test plan ap- 
proved for the project involving a variety of experimental 
and control groups and including highly specified procedures 
to be followed by experimental groups. 

P-24 List of Field Test Sites . A list of approved and cooperating 
schools prepared and negotiated by the Project Directors. 

The list was generated by following criteria constraints as 
to geographic dispersion, district size, minority culture 
distribution, etc. 

P-25 List of Consultants (Confirmed) . A list of consultants 

identified, negotiated and approved by the Project Directors. 
The list was generated by following criteria constraints as 
to institutional affiliation, professional discipline, area 
of expertise, etc. 

P-25 Proposal . The contractual paper upon which the project was 
funded and operationalized. It was generated through the 
cooperative effort of the six Directors affiliated with the 
English departments of the participating universities. 

P-15 Journal Report of Project Progress (Project Director) . 

A report prepared by the senior initiating Project Director 
expressing the view and progress of the project and published 
in an English discipline professional journal as one of 
several inclusions focusing on the application of performance 
objectives technology to the teaching of English. 

P-04 Field Test Data from Use of Preliminary Catalog . The 

collection of data derived from field test use of the catalog, 
including responses to questionnaire, interview reports and 
record forms, and anecodotal comments made by teachers in 
the form of marginal notes in second copies of the catalog. 

P-06 Teacher Annotated Catalog . One of two sets of catalogs 
given teachers, returned to project staff by them with 
annotated comments, critiques, and suggestions resulting 
from its use. 

P-08 Interviewer Written Reports . A written report made by 

Project Directors on the basis of site visits to schools 
late in the field test period. The reports followed a 
somewhat standard format, with information obtained during 
each interview summarized in narrative form. 

P-12 Video Tape Critique of First Draft of Performance Objectives . 
A video tape recording of one of the two major consultants 
critiquing the draft of the catalog prepared following the 
first working conference. 
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P-16 Dissenting Position Paper (Journal Article) . A paper pre- 

pared by one of the consultants participating in the 
initial working conference. It set forth a dissent in 
relation to the intents and objectives of the project. 

The paper was one of several inclusions in an English 
discipline professional journal, focusing on application 
of performance objectives technology to the teaching 
of English. 

P-17 Response to Dissenting Position Paper (Journal Article) . 

A paper prepared by one of the Project Directors in 
response to a dissenting paper presented by one of the 
consultants. It was designed to place into project per- 
spective the points of dissent and was published as one of 
several articles in an English discipline professional 
journal focusing on application of performance objectives 
technology to the teaching of English. 



Products serving project production : 

P-20 Outline of Catalog Goals > Objectives and its Organization; 

First Draft . A planning outline setting forth in oper- 
ational terms the products to be produced for inclusion in 
the catalog. Created by the Project Directors as a frame- 
work around which the initial working conference could 
be organized. 

P-21 Examples of Performance Objectives . An initial set of per- 

formance objectives created by the Project Directors to 
illustrate a manner in which performance objectives might 
be stated. They were designed as specific items around 
which participants in the initial work conference could 
react for the purpose of determining the desired level 
of objective specification to be used in the catalog. 



Event serving a project management function: 

E'-30 Field Use of Preliminary Catalog . A field test activity 

requiring the use of the catalog in English instruction 
in 24 schools around the United States. The test was 
designed to permit natural, noncoercive use of the 
catalog by teachers, with data to be. collected around 
the reasons for use or non-use. 

Events serving project production functions: 

Conference — Field Representatives . A conference held to 
familiarize representatives from each of the field test 
schools with the contents, intents, objectives, and uses 
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of the catalog of performance objectives. The purpose 
of such familiarization was to facilitate distribution 
and use of the catalog in the various sites and to increase 
coordination in data-gathering efforts. 


E-28 


Initial Work Conference. (October, //I) A conference of the Proiect 
Directors and their consultants in which the purposes were, 

( 3 ) to give substance to the initial goals and ob- 
jectives suggested by the Directors, and (b) to begin 
the task of generating a broad range of performance 
objectives considered relevant in the teaching 
and learning of English. 


E-29 


Second Work Conference. (March, //2) A conference of the Project 
Directors and consultants in which the purpose was to 
react to the work of the Directors in assembling, organ- 
izing, and modifiying the outputs of the initial con- 
ference, and to continue the task of generating and 
framing a reasonably representative set of objectives 
for Grades 9-12. 



Output map . Figure 4 illustrates the interdependencies of the 
various outputs of the BOE Project to the extent identified and under- 
stood by the site visitation team. It is probable that one could 
intuit and even verify the existence of a larger number of outputs 
depending on the level of detail useful in understanding the project and 
the nature of its outputs. In the present instance, the intent has been 
to illustrate those identified in a manner as free from the time dimen^ 
sion as possible so as to communicate the functional influence of the 
outputs on each other. 
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Chapter III: Sunmary of the Data 



Data were gathered around specific outputs selected from those 
described in Chapter II. The interviews sought to elicit for each out- 
put to be analyzed the standards by which the satisfactory completion of 
the output is judged, the tasks required to generate an output meeting 
those standards, and the enablers (knowledges, skills, and sensitivities) 
which facilitate the performance of that set of tasks. Presented first 
is a summary discussion of each of the major category sets of data, i.e., 
standards, tasks, and enablers, followed by a series of tables which pre- 
sents the frequencies with which the various categories of statements 
were made within each set.^ To establish the parameters of the data 
gathered, the following outputs were selected for interview and further 
examination. (The identification numbers correspond to those used in 
all tables in this profile.) 

P-01 Final Catalog of Performance Objectives in English, Grades 9-12 
P-02 Commercially Published Edition of Performance Objectives in 
English, Grades 9-12 

P-04 Field Test Data from Use of Preliminary Catalog 
P-07 Interview Questionnaire (Guide to Interviewers) 

P-09 Preliminary Catalog 

P-17 Response to Dissenting Position Paper (Journal Article) 

P-18 First Draft of Introduction to Catalog 

P-20 Outline of Catalog Goals, Objectives and its Organization: 

First Draft 

P-25 List of Consultants (Confirmed) 

E-27 Conference, Field Representatives 
E-28 Initial Work Conference (October, //I) 

When one or more of these outputs are missing in a table, it simply 
means no data were collected in that category. It is important to re- 
iterate at this point that the BOE Project staff consisted of seven Project 
Directors with no additional professional staff. Thus, interviewees were 
in a position to provide first hand data across both production and 
management functions. 



O 

''If the reader is interested in the narrative statements of the 
interviewees (raw data), these can be found in Che Appendix. To locate 
Che narrative statement for any given category, first note Che output 
and its identification number in Che Cable. Second, note Chat each 
descriptive label within a given category has a distinct number or code. 
Turn Co Che Appendix and locate Che output. Under Che output locate Che 
category label or heading (standard. Cask, or enabler) and pinpoint Che 
number or numbers (depending on frequency cited) of the descriptive label 
which appeared in Che Cable. The statement in Che Appendix opposite this 
(these) number (s) is Che original narrative statement being represented 
in Che Cable. 
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Standards Held for Outputs 

The statements elicited from interviewees were somewhat varied, but 
patterns emerge when one examines them in light of the range of standards 
) cited. In terms of standards relating directly to output quality (Table 

2), a broad range of standards categories were cited for two outputs 
(P-09 and P-17), while the rest were cited in only one or two categories. 
Completeness, utility, and acceptability of outputs, ranging across a 
variety of outputs, emerge as the prevailing themes. Frequencies relating 
to process standards held for operational matters around each output 
(Table 3), in some Instances, have applicability only to the outputs in 
which the citation occurs. In summary, however, the concern indicated 
Involves the utilization of the services of others, achieving favorable 
response from them, and producing a product that has utility and will be 
used. (See the Appendix for related raw data.) 



Tasks Pertaining to Output Attainment 

A total of 150 task statements were elicited from interviewees 
relative to 11 outputs. These statements divide almost equally between 
direct, produclng-types and those generally thought of as management. 
Table 4 indicates the frequencies of citation across a broad range of 
task categories for each output. Significant clusterings appear around 
tasks relating to: 

(a^) clarifying problem addressed 
(^) producing the output 
(c) assessing the output quality 
(^) procuring systems/services 
(j^ facilitating relationships. 

Further examination of Table 4 will give the reader an impression 
of categories of tasks clustered within outputs, and the range of 
categories cited which are specific to each. The larger total frequency 
of citations for the preliminary catalog is a function of data being 
gathered from two Interviewees. However, the distribution of citations 
for this and other outputs suggests attention to detail on major critical 
outputs of the project. 



Enablers Pertaining to Output Attainment 

Consideration of enablers is in relation to categories of knowledges, 
skills, and sensitivities. A brief discussion of the citations within 
each of these categories is called for, followed by a discussion of the 
interrelationships of the data within the general enabler set. 

Knowledges . The most frequently cited knowledge factors involved 
in the BOB Project (Table 5) occurred in categories where one must examine 
the raw data for the most useful Information (see Appendix). Briefly, 
knowledge requirements clustered around the nature of English and its 
teaching, and, in general, situational factors in an educational environ- 
ment. 
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Skills * Table 6 presents the frequencies with which the various 
skill categories were cited for the outputs analyzed* Skill enablers 
were mentioned for seven of the 11 outputs for which data was sought* 
These clustered around communication, l)oth writing and interpersonal, 
and self-discipline factors* The total number of responses in this 
category was admittedly small, but a possible significance of the data 
will be considered further in discussion of the interrelatedness of 
the enabler citations* 

Sensitivities * By far the greater number of enabler citations 
occurred in the sensitivity category* Loadings appear (Table 7) 
around the capabilities and limitations of others; 

sensitivity to the worth of the project, as well as to existing value 
systems which interact; and to potential communication barriers* The 
range and frequency of citations was highest in Output P-20, an output 
which constituted the first identifiable pulling together of the cata- 
log* 



Interrelationships among the enablers * Emphasis on knowledge factors, 
particularly in English, a broad range of sensitivity factors, and a 
relatively narrow set of skill factors seems consistent with the organi- 
zational anl staffing structure of the project* To engage in producing 
a catalog of performance objectives in English one must have knowledge 
of the discipline, as did six of the seven Directors* In light of rela- 
tively limited staff expertise in framing performance-type objectives, 
there was a need for major consultant help from persons outside the dis- 
cipline, This factor, coupled with opposing views within the discipline 
regarding the application of behavioral technology to the teaching of 
English, would seem to explain the emphasis on sensitivity* 



Discussion of the Output Data 

The character of the data gathered by the interview team appears 
consistent with several significant impressions gained from being on- 
site* Extended discussion of these impressions appears in Chapter V* 

For the present purpose, however, it is again noted that the project 
staff consisted of seven Project Directors. Differences in project 
responsibilities were around substantive catalog content, rather than 
management decision lines* Thtrs, the data reflect a view of a project 
through the eyes of a manager-producer. Additionally, unanticipated 
outputs occurred which focused public attention on the efforts of the 
project and resulted in the exercise of management responsibilities and 
tact that otherwise might not have occurred* (These were the papers in- 
volved in the dissent position*) 



In terms of the relatedness of standards, tasks, and enablers, it appears 
that quality, utility, and acceptance were the standards sought for the cata- 
log and the tasks seen as consistent with getting there* Appropriate and 
effective utilization of resources as a process standard seems in accord with 
efforts to clarify any problems and to effect a supportige field setting* With 
the exception of knowledge of the English discipline, most enablers appear on 
the affective side, with probable cause. It is possible that without maintaining 
a ”tuned-in*^ posture and avoiding over-response to the various situations which 
arose, no amount of technical skill could have naintained the essential in- 
tegrity of the project* 
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Chapter IV: Supplementary Data 



Additional data with respect to this site visit comes from a question- 
naire technique as well as from orderings of all the outputs identified 
in accordance with various categories in which they were classified The 
sections to follow include: (^) classifications of the output character- 

istics, (^) summary of staff background, (jc) summary of position require- 
nents and support systems, (jd) summary of the significance of various 
general categories of work, ie) summary of project funding, (_f) discussion 
of supplementary data • 

Classifications of Output Characteristics 

Outputs may be categorized in terns of a number of variables* Among 
them are: {j) Structure (product, event, or condition), (^) Function 

(policy setting, management, or production), {c) Level (focal, component, 
or facilitating), (j^ Character (loiowledge, technology, implementation, 
or information), and (^) Stage of completion. These five schema are 
represented in Table 8 for each project output identified, with frequencies 
summarized for each category. 



Summary of Staff Backgrounds 

Personnel data were obtained from six of the seven Project Directors. 
Of these, three hold doctoral degrees and three hold master's degrees. 

Major areas of specialization include four specializing in English/writing, 
one in journalism, and one in education/ teaching. All respondents are 
members of the National Council of Teachers of English. Three have 4-8 
publications, while the others have over 16 publications each (one 
includes 12 books and over 100 articles and reviews). 

In terms of work experience, the Project Directors reflected years 
of experience in educational RDD&E ranging from 3 to 25, with 0 to 11 
of these years reported as engaged in directing such efforts. College 
or university teaching experience ranged from 3 to 35 years while 
experience in public schools varies from 2 to 18 years. 

The most frequently cited aspect of professional training considered 
relevant to this project related to training in English and education. 

One respondent felt that research courses taken in an educational psy- 
chology nrogram were most relevant. Work experiences in English curri- 
culum studies and the current project were the kinds of experiences 
considered most relevant. 
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Summary of Position Requirements and Support Systems 

Of the various aspects of training and work experience backgrounds 
of the project staff, there was consensus on relatively few with respect 
to their importance as requisites. Knowledge and skill > coming from 
both formal training and work experience, in the content and teaching 
of English was considered essential by all. To a somewhat lesser degree, 
a sense of educational psychology and psychology of learning was con- 
sidered important. In terms of number of years of training and related 
work experience, responses were as varied as academic backgrounds. 

By and large, manpower and other material resources were considered 
by project personnel to be adequate. Time lines and financial resources, 
on the other hand, were considered by most to be **a bit tight." In spite 
of this limitation there was general satisfaction that sufficient project 
time was provided for professional presentations and for professional 
perusal of literature. 



Significance of Various Categories of Work 

Table 9 reflects the frequencies with which the Project Directors 
ranked nine general categories of work across eight levels of signifi- 
cance. The scale used for the ranking and shown in the table is as 
follows: 

0. Definitely not a part of my project activity, does 
not apply. 

1. Under unusual circumstances may be a minor part of my 
work. 

2 . 

3. 

4. A substantial part of my work. 

5. 

6 . 

7. A most significant part of my work. 

The table reflects concurrences on significance and common responsibil- 
ities in relation to the writing task, perhaps the single most essential 
of the project. However, the balance of the table, when taken in con- 
junction with factors noted onsite by the interview team, indicated 
each Director undertook greater responsibilities for some general cate- 
gories of tasks than did others. While project management decisions 
were generally made by the group as a commonly shared responsibility, 
individual Directors did assume various initiating and leadership roles 
for discrete tasks, with production or consensus support coming from 
the rest. 
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TABLE 9 

General Activity Significance 



General Activities 






Levels of 


Slenlf Icance 
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1 


2 
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5 


6 
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Reading 










3 






3 


Designing/planning project 
procedures 








1 


3 




1 


1 


Developing research tools, 
data Instruments 




3 


1 




1 


1 






Collecting project data 










5 


1 






Analyzing data 




2 






3 


1 






Writing 












1 




5 


Supervising/ coord ina ting 


3 






1 


3 






2 


Teaching or training 


3 






2 


1 








Meeting / consul t ing/ advsg 








1 


1 






A 


Totals 


6 


5 


1 


5 


20 


A 


1 


15 



Summary of Project Funding 

The Tri-University BOE Project is a two year project funded totally 
by the U.S. Office of Education. Allocations for each of the two years 
were as follows: 

June 15, 1969 - June 15, 1970 $125,000 

June 15, 1970 - June 15, 1971 $125.000 

Project Total $250,000 
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Housing for the project with an estiiTiated value of $14,400 was provided 
by each of the three universities. At the time of site visitation a 
two month extension of time, with no extension of cost, was granted 
the project to complete catalog preparation for publication and final 
reporting tasks. Extended terminal date for the project was scheduled 
for August 15, 1971. 



Discussion of the Supplementary Data 



Genrally, the supplementary data indicates a staff exceptionally 
well versed in the discipline they represent, i.e., English and its 
instruction. A good deal of experience was also represented in terms 
of RDD&E activities. Project staff is seen as having organized around 
equal directorate responsibility, but did divide tasks presumably along 
the lines of expertise or accessibility of the resources, etc. 
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Chapter V. Project Dynamics 



The Tri-University BOE Project, in proposal terms, appears as 
a relatively straightforward developmental effort using consultants 
as the primary source of content input in building a product, and re- 
fining the product for publication. However, organizationally and 
operationally the project reflects several dimensions and features 
which set it apart from other projects described in this series; some 
of these have been alluded to in previous chapters of this profile. 

Worthy of extended discussion are: (a) the operation of the project 
staffing structure; (_b) the issues raised; (O content vs . technological 
foci; (d) implications of the dynamics in the evolution of the catalog. 

Operation of the Project Staffing Structure 

Consideration of project dynamics as influenced by the staffing 
patterns employed focuses on three structures, the Tri-University 
relationship, the director relationships, and consultancy factors. 

Tri-University structure . The collaboration of the three univer- 
sities in the design and conduct of this project had as its background 
the history of English departments pursuing, on national and regional 
scales, the improvement of instruction in reading and English related 
curricula. Each was reasonably acquainted with the interests and di- 
rections of efforts of the other. In terms of these directions and 
interest, the proposed effort was seen as filling one of many needs 
identified by each of the institutions, i.e., bringing clarity to the 
objectives one might use for English. While discussed more fully 
in the ^issues** portion of this section, it must be noted that the 
same sense of project utility was not shared by all colleagues 
within each of the English departments. However, there was general 
consensus **in principle” among colleagues that a need existed and 
that the present effort could result in a useful product. 

As a matter of administrative convenience, the University of 
Illinois was the contracting institution, paying the bills and monitor- 
ing expenditures of funds. In practice, federal funds for salaries of 
directorate personnel from the other two institutions were paid directly 
to those institutions for subsequent disbursement . Royalties subsequently 
accruing from the project are to be handled in like manner. 

Relationships among the directors . The initial proposal was 
submitted under the formally declared directorship of six persons, two 
each from the three universities. Subsequent to submission of the pro- 
posal, an amendment was submitted which, among other revisions, added 
to the directing staff an educational psychologist from Purdue Univer- 
sity. With a background of the principles of learning as they might 
apply to reading and audio-visual communication, the additional Director 
was perceived as possessing certain psychological and research skills 
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needed by the project. As the project got underway, each of the Direc- 
tors assumed leadership responsibilities for various content specific 
areas of the subject matter to be generated for the catalog of objectives 
in English. The actual division of such production labor was according 
to preference and/or areas of expertise and familiarity. Where overlap 
occurred by way of common concern or skill, such Directors as were in- 
volved became an interacting group in support of the Director assuming 
leadership for that particular section. 

On the management side of operations there evolved further divi- 
sions of labor (the actual exercise of various responsibilities) to 
facilitate accomplishment of overall administrative arrangements, e.g., 
conferencf ^2 arrangements. These tasks were assumed primarily by personnel 
from the University of Illinois (the contracting institution). Beyond 
purely administrative matters, management concerns arose as a function 
of personnel operationally confronting issues inherent in the effort. 
Leadership on this matter evolved to the senior initiating officer from 
the University of Illinois who was seen as possessing the skills and 
sensitivities necessary for maintaining the integrity of the project and 
a sense of perspective among its participants. 

The consultants . Three major levels of consultancy are apparent 
in the project. One level relates to the inhouse consultants identi- 
fied in two of the participating institutions. These consultants parti- 
cipated in the work conferences conducted and were readily available to 
the Project Directors for consultation between conferences. The main 
consultant group was comprised, by design, of people from across the 
United States who were selected on the basis of contributions they could 
make to the work at hand. Public school representation was essentially 
confined to persons working in the settings to be used in the field test, 
particularly those settings selected from those participating in the 
ES-70 program. The ES-70 focus was in keeping with terms set forth in 
the "Request for Proposal." Finally, two consultants served as techni- 
cal advisors to the project in terms of applying the technology of per- 
formance objectives to the content area. Somewhat diverse consultant 
backgrounds or opinions relative to the content area was sought, and to 
some extent was achieved. The outcome, given the above structures, set 
the stage for confrontation around "means" issues in designing English 
curricula which consumed significant portions of project energies. 



The Issues Raised 



Central to the issues raised was the appropriateness of specifiying 
performance objectives for a content area such as English. While it was 
intended that the first work conference direct attention to the genera- 
tion of such objectives, the diversity of backgrounds and biases held 
by participants came into play regarding whether or not the assigned 
task was both appropriate and possible to accomplish in any meaningful 
sense . 

Out of the conference came two significant outputs, an initial set 
of statements describing objectives for English (in crude ind somewhat 
random form), and a dissenting position paper prepared by one of the 
consultants. The dissenting position paper presenting, essential.^, 
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the views being expressed that ran counter to project objectives. This 
paper prompted a subsequent response prepared by one of the Directors, 
Both of these papers, along with a project progress report prepared by 
the senior initiating officer and a position statement by the sponsor’s 
monitor were subsequently published in an English professional journal. 

While project staff had been aware of the tentativeness of col- 
league approval regarding behavioral objectives for English prior to 
the first conference, they became more acutely aware of it afterwards. 
Some of the side effects included a reticence on the part of some project 
staff members to discuss the project efforts with outside colleagues. 

In addition, at least one cluster of the main consultant group attending 
to the specific content of one of the chapters of the catalog continued 
diligent work on the material but declined the invitation to be listed 
as contributors to the catalog. This appeared to be as a result of the 
controversy which arose over whether behaviorally framed objectives are 
appropriate for a content area like English. In short, the professionals 
in English had come to a public standoff within the discipline, with the 
Tri-University BOE project as the backdrop. The effects of this on 
project staff are discussed in the final section of this chapter. 



Content vs . Technological Foci 

While it may be argued that a behavioral technology intrudes on a 
discipline historically treated as relating to the humanities (an area 
presumably impossible to reduce to behaviors in any ’’appropriate sense”), 
the project personnel were directed to manipulate the technology in 
keeping with the discipline. Admittedly the tool to be employed was 
central to the project intended, but one cannot say the tool or the 
directorate coerced efforts contrary to professional conscience . In 
fact, concerns expressed by those participants remaining (at least one, 
the author of the dissent paper, left the project) seemed more focused 
on what ’’people might think” rather than on any gross violation of 
principle. The application of the technology was conceded, in the dis- 
senting paper, as ”a helpful thing” in terms of an ’’exercise in clear 
thinking” for those tending to be more ”f uzzy-headed” about what they 
are doing than are math or science teachers. Without disparaging either 
point of view in the least, it was observed by the interview team that 
there was at least a healthy exercise in clear thinking regarding the 
teaching of English, And just as the dissent points to the incommen- 
surables surrounding the learning of a more taciturn student, the long 
range contributions of this exercise may never be fully understood. It 
is probable, however, that negative side effects will be minimal since 
the users of such products tend to shunt them to the shelf at the first 
sign of difficulty. Project staff gave evidence of understanding this 
phenomenon in their direct, open-ended effort to collect use-nonuse data. 
It was in this context that project staff persevered and the final 
shaping of the catalog neared completion. 



Influence of Project Dynamics on Evolution of the Catalog 

Early in the history of the project, even prior to actual funding, 
project staff were sensitive to the potential issues involved in the 

iaIa 
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effort. To counter the fear that there might be a tendency to use the 
catalog in a manner that suggested all objectives must be behaviorally 
based, an early modification to the title was made with the addition of 
the word ’^representative*' (behavioral objectives) . The concept of the 
"open-endedness" of the catalog was introduced. As the project began 
operations and neared the conclusion of the first year, another title 
modification seemed indicated and was acted upon on the advice of the 
major consultants. To reduce the negative affect surrounding the catalog 
(and possibly as a function of liberties taken with the technology) the 
term ’’behavioral" was replaced with "performance." 

In an effort to create objectives having a relatively obvious and 
direct relationship to traditionally abstract gcal statements, the 
measures of objective achievement which are typically explicit in the 
pure behavioral objective form were permitted by staff to become more 
and more implicit. As a consequence many performance objectives were 
structured in which the criteria for judging tlieir successful completion 
was left to the standards held by the individual teacher. It is possible 
that this adaptation of the technology "unhooked" many of the objectives 
writers and permitted, in fact, a more comprehensive coverage of the 
discipline than might otherwise have occurred. 

Finally, in terms of catalog content, the issues raised and the 
adaptations of the technology employed gave additional structure to the 
section of the catalog that describes its purposes and use. The evalua- 
tion factor, the measurement unit to be used by teachers as acceptable 
indicators of objective achievement, is considered at length. A caveat 
is entered to the effect that the teacher must exercise discretion and 
skill in seeking out observationally the subtle indicators of acceptable 
change which marks the covertness of learning among so many students. 

Perhaps the impact of the dynamics of the project was felt most 
heavily by the directorate. In a real sense the size of the directorate 
was probably fortunate in that the pressures of the project could be 
distributed beyond just one or two persons. By and large, the Directors 
were committed to responsible, quality completion of the project task, 
even though several had little background in the behavioral sciences, principles 
of learning, and, indeed, the technology of behavioral objectives. For 
those lacking such a background and/or a sense of confidence in supporting 
the effort, the first year represented a period of intensive personal 
study and evaluation of related literature . As the project moved through 
time and the technology became more suitably adapted to the present ef- 
fort, a growing confidence in the viability of the concept and an in- 
creased awareness of its potentials occurred. The staff of six English 
professionals and one educational psychologist emerged in the project 
with a verbalized sense of the real contributory and complementary nature 
of other disciplines. 

The obvious concern for the utility of the catalog, its favorable 
acceptance Ly peers, and the need for broadly based contributions from the 
field, as mentioned in Chapter 111, takes on a new perspective in light 
of the dynamics in operation within the project. The challenge to 
management in channeling such interactions aiong productive lines and 
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maintaining participant focus on the end goal was obviously difficult. 
That the challenge was met and the goal, for all practical purposes, 
reached, was attributed by the directorate* with complete unanimity, 
to the “inexhaustible** patience, tact, diplomacy, and sense of fair 
play displayed by the senior principal investigator. 
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Chapter VI: Implications for Training 



Examination of the raw data on enablers in the Appendix provides 
a sense of the prerequisites for engaging in a project structured and 
organized along the lines of the BOE Project. Beyond the expected 
emphasis on content specific knowledges and sensitivities, the requisites 
read not unlike requisite data which might be coxlected from other projects. 
What is missing is an indication of the affect with which the requisites 
were stated and the level of significance given specific skills relative 
to the challenge presented the staff. 

Several themes having relevance for the training task seemed 
to emerge from analysis of interview protocol and interview team impres- 
sions of this site. One instance is the realization on the part of 
project staff that the technology of behavioral objectives provides a 
tool to achieve certain ends, and that as a tool its utility depends 
upon its adaptability to the varying situations in which application 
is attempted. It seems appropriate, then, to suggest that training in 
such techniques emphasize not the technique itself, but the principles 
upon which it is based and the processes and conditions under which the 
tool, rather than the environment, might be shaped. 

Another factor which comes to the surface in this project has to 
do with the cognitive development of skills in RDD&E and the degree 
to which those skills can be processed, translated, and generalized when 
working and communicating with naive colleagues. The specific content 
involved varies greatly, e.g., from questionnaire construction to 
management of dissent to maintenance of goal orientation with tact 
and diplomacy. One suggestion offered by a Director emphasized training 
which included placing the student on the receiving end of the service 
for which he is being trained. The intent of such training is to provide 
experiences in which the student might internalize, beyond the purely 
cognitive level, the meaning of that which he is being taught. For ex- 
ample, the principles of questionnaire construction might well be taught 
as basic mechanics, i.e., the tools for building the tool. However, 
before, during, and after such trainin'^ the real emphasis should be on 
the nature of the crucial information that is being sought and the vary- 
ing ways in which it is expressed. 

There is a strong indication that training must include 
the building of general skills in appropriate problem identification, 
i.e., the ability to concisely isolate those problems which influence, 
shape, or are the foci of goal orientations. Further, it is indicated 
that once a problem is identified, the worker must have the capacities 
and characteristics which enable him to assume an actively passive role 
in the situation. Such a role Implies that educational RDD&E efforts 
are, in fact, helping services which command a set or sets of useful 
tools, and that such services cannot be effectively rendered until 
members of the serving staff, as well as those being served, achieve 
mutual commitment to the goal and the implied means and degrees of 
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freedom to deviate are agreed upon. By remaining actively passive in 
the situation, the worker receives and processes a broader range of 
information regarding the various problems and is able to more 
precisely place that information into a perspective which enables 
all concerned to address and resolve the real Issues. 

The relative lack of experience of the majority of the project 
directors in the area of technological expertise with regard to 
behavioral objectives appeared not to prove a handicap in the project. 

This factor seems to have been compensated for by the fact that all pro- 
ject staff had a clear understanding of the ultimate objective (the focal 
output, if you will) which guided them in decisions regarding consultants, 
content, work responsibilities, etc. This indicates an Important conclu- 
sion with respect to training: l.e., if the goals of a project are clear 
to all staff, then the appropriate expertise can either be acquired by 
staff or brought in from outside to perform specific tasks essential to 
the successful accomplishment of those goals. 
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Appendix: Listing of Output Standards, Tasks, a’ld Enablers 



The following is a list of standards, tasks, and enablers for out- 
puts around which interviews were conducted. These statements were ex- 
tracted from discussions with interviewees and were coded into their 
respective category sets. The selected code precedes the statement 
and indicates the following for: 



STANDARDS 



Code J: 
J-1 
J-2 

Code LM: 



Structure of Standards. 

Standards against which outputs are judged. (output oriented) 

Standards against which processes and/or operations are judged, 
(process oriented) 

Primary Categories of Standards. 



TASKS 

Code NO: Clusters of Tasks. 



ENABLERS 



Code S: 
S-1 
S-2 
S-3 

Code UV: 



Structure of Enablers. 

Knowledge. 

Skill or ability to perform. 

Sensitivity or awareness. 

Primary Categories of Enablers - (knowledges, skills, or 
sensitivities) . 



The codes associated with these three categories (standards, tasks, 
enablers) are the same both here in the listing and as previously cited 
in Chapter III tables. 

Each of the 11 analyzed outputs is cited below within a rectangular 
box. Listed under each are the interview statements relevant to that output. 

P-01: Final Catalog of Performance Objectives in English, Grades 9-12^ 




STANDARDS: 

J m 

2 34 Students reflect positive attitudes toward literature in the 

process of accomplishing the performance objectives. 
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1 22 
2 17 



TASKS ; 

NO 

05 

03 

06 

06 

06 

06 

06 

06 

06 



Teachers use and expand on the activities suggested by the 
performance objectives, (as planned). 

The profession (English Education) ultimately judges the cata- 
log to be a service rather than a disservice to students and 
the profession. 



Review field test data (summarized questionnaire, catalog anno- 
tations. site visit reports, etc.). 

Examine sequencing of performance objectives in light of avail- 
able taxonomies of learning. 

Revise chanter titln (Reading and LiteTature) to reflect more 
directly tnat the contents are perfoiTnai.ce objectives (Respond- 
ing to Literature) , 

Reorder objectives to suggest free-wheeling (student likes and 
dislikes) as the most appropriate beginning activities. 

Reorder objectives to suggest analytical and critical, response 
items as being at *:eriuinal or end points in the leamers devel- 
opment. 

Insure adequate numbers of '^exposure** and ‘‘interpretation" 
items in catalog chapter to provide a basis for subsequent 
(cognitive) understanding of the contents of literature. 

Provide for expanded suggestions for and uses of creative 
experiences (in learning about literature) e.g., scripting 
and performing. 

Add contributions coming from teachers in relation to more 
current (or relevant) pieces of literature as the media for 
a given performance objective. 

Check objectives against their ability to serve at least the 
75% of the population having reasonably functional reading 
levels . 



ENABLERS : 

S W 

2 01 Experience in secondary school classrooms with a sustained 

and ci’rren.': contact vrith classroom. 

2 02 Must De able to exercise tact. 

2 02 Must br able to relate to people very well. 

3 16 Must be somewhat free of academic preoccupation. 

3 16 Must be polite. 

3 Must have a sense of "team effort." 

3 22 Must be sensitive to the Impact of field test activities on 

the field test sites. 

3 11 Sensitive to not "locking student? in" (provide alternative 

response modes). 

3 31 View the effort as one of providing resources for learning 

as opposed to resources for teaching. 

3 16 (Sensitive to) behavioral objectives being alien to the 

discipline of English. 

3 16 (Sensitive to) the English profession being threatened by 

behavioral objectives . 

^•-’83 
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P-02: 



Commercially Published Edition of Performance Obiectives in Enelislu 
Grades 9-12 ^ 



STANDARDS* 

J Ui 

1 32 Publishing house selected is an older, better established 

publishing house. 

1 32 Publishing house selected already has a reputation and is 

known by teachers. 

2 36 Publishing house selected normally publishes English texts 

and related materials. 

2 36 Publishing house selected normallv publishes materials 

directed to high school students. 

2 36 Publishing house selected normally publishes materials 

directed to high school teachers. 

2 36 Publishing house selected has a large, successful, well 

staffed sales force. 

TASKS: 

m 

05 Submit statement to OE in support of negotiated contract 
with publisher as appropriate. 

06 Revise letter requesting a statement of publisher interest 
as directed by and/or incorporating OE suggestions. 

07 Describe to potential publishers the project and kind 
of book that is expected to emerge. 

23 Write a letter requesting a statement of interest in 

publishing the document direc ed to at least three publishers. 

23 Submit to publisher selected a model, OE approved contract. 

23 Work out with the publisher the problem of contracting 

in a manner different from the publisher’s usual contract. 

23 Obtain contract approval from authorities in the agency 

(university as contracting age^ for the project). 

23 Meet with project directors to consider publishing houses 

to be approached. 

23 Consider the publishing houses that meet the criteria, 
standards set forth. 

24 Obtain OE approval of the publishing houses from whom 
statements of interest will be sought. 

24 Submit most favorable publisher interest response to OE 

for approval. 

24 Submit a draft of letter requesting a statement of publisher 

interest to OE for approval. 

24 Submit proposed contract negotiated with publisher to OE 

for approval. 

29 Submit a copy of field test version of catalog to publisher 

for examination. 
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ENABLERS: 



J 

1 12 
1 01 

TASKS : 

NO 

29 

29 

29 

29 

29 

05 

05 

22 

04 



Knowledge of publishing houses having a reputation for 
publishing English tr* ‘is and materials for high schools. 
Knowledge of publishing sales forces, their size, scope, 
reputation and effectiveness. 

Knowledge of OE regulations and procedures governing 
publication of OE suppcited project documents. 



Test Data from Use of Preliminary Catalog 



100% return of the questionnaire (required by the nature of 
the product and the field test). 

Each of the questions — determined earlier and for which data 
was collected to answer — have been addressed. 



Send questionnaire to liaison person at each field test site 
(school) as an example of what is to be sent to all teachers 
participating in the field test. 

Send questionnaires to liaison person to distribute at the 
field test site to all teachers participating in the project. 

Send out a reminder letter (two weeks after the initial 
marking of questionnaires for the teachers) explaining the 
id for this information. 

id out a second reminder letter with another set of question- 
naires (one week after the first reminder letter). 

Call the liaison person and earnestly request the questionnaire 
information be sent as soon as possible — this was for personnel 
in a particular school who had not responded to the 
questionnaire after the second reminder letter. 

Record all marks and comments from all of the annotated catalogs. 
Arrange catalog annotations in relation to each objective - 
so the author revising a specific objective may be made aware 
of all the reactions to that objective from those who annotated 
the catalog. 

Adopt a **hands off” policy relat:' /e to teachers annotating 
the catalog — they were given no instructions as to how to 
annotate; they were merely invited to make annotations. 

Assemble information from field-site interviews into a 
written report descriptive of some of the conditions under 
which the catalog was used. 
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05 Tally and assemble all infoimation from the questionnaire 

by each question. 

05 Assemble demographic information in general manner to indicate 
the breadth of the sample and school situations in which the 
catalog was tried out. 

06 Edit data analysis report so that it will be more readable (under- 
standable) for the intended audience . 

ENABLERS: 

1 uv 

1 08 Know what information is wanted from the data. 

1 08 Know what questions are to be answered from the field test 

data/inf ormation. 

2 39 Ability to develop the most direct and least biased method 

to elicit the information. 

2 11 Resist the temptation to develop a new scale (or number of 

scales) for some situation-specific thing. 

2 11 Resist the tendency to gather more data than serves the 

immediate specified purpose. 

2 14 Ability to write (report) in a clear and unpretentious 

manner. 

3 26 Realization that missing data (from people who have not 

responded to the request for information/data) is often 
the most important. 



P-07: Interview Questionnaire (Guide to Interviewers) 



STANDARDS: 

J m 

1 05 Information from the interview (following the interview guide) 

will be useful for making decisions relative to the 
catalog revisions . 

TASKS; 

m 

01 Determine the types of information which would be useful to 
revise/upgr^de the catalog. 

30 Determine sources of people that would provide the moot 

relevant information in the school setting. 

03 Consider how to elicit information without biasing the 
situation. 

02 Consider the functions to be served by the interview situation. 

04 Generate/develop a set of questions for the guide. 
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04 


Arrange questions (the order of) so that the initial questions 
are the most general and least “loaded”. 


24 

06 

03 


Review the initially generated set of questions. 

Modify questions to clarify them and make them less ambiguous. 
Consider the language — terminology — in framing questions, 
i.e., avoid references to psychological terms that imply 
a behavioristic approach to psychology. 


03 


Consider the function of the interview guide — to serve as a 
suggested set of questions and not as a fixed entity. 



ENABLERS 



s uv 
1 08 
1 08 


Know the purpose of the interview. 

Know what information is not on hand but would be needed and 


2 35 


useful to revise the catalog. 

Ability to phrase question in such a way that it isn't loaded 
and doesn't imply a value judgment. 


2 35 


Ability to phrase a question so that it does not imply that one 
answer is right and another is wrong. 


3 24 


Sensitive to the notion that an interview situation is open 
to all kinds of bias and that some sort of interview guide 
is necessary to insure that the interviewer elicits the pre- 
scribed kind of information rather than reacting to and pursuing 
a bias. 


3 18 


Sensitive to the possibility of defensive feelings on the part 
of public school personnel relative to university people and/or 
personnel involved with national projects. 


3 16 


Awareness of the issues and some of the hostilities relative 


3 30 


to the notion of behavioral objectives from people working 
in the field of English. 

Sensitive to the audience for whom the questionnaire is 
being developed. 



P-09: Preliminary Catalog 

STANDARDS : 



J LM 
1 16 


Objectives compare favorably with other known documents or 
expressions in the literature. 


1 05 


Broad use of the catalog or its components occurs during 
fieL' test. 


1 13 


Expression of consultgi^ satisfaction with the objectives. 
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TASKS: 

m 

29 

29 

29 

29 

29 

02 

04 

29 

06 

24 

24 

06 

29 

24 

06 

06 



Get working consultants acquainted with each other in an 
initial working conference. 

Explain to consultants in the initial conference why a 
nonverbal communication section to the catalog should 
be written. 

Dramatize (an illustrative) set of nonverbal signals 
(associated with communication) to the consultants. 

Elicit from the consultants other examples of nonverbal, 
communicating behaviors. 

Engage consultants in formulating instructional goals relative 
to the area of nonverbal communication. 

Participate in group effort at explicating instructional 
goals in the area of nonverbal communication. 

Formulate and write out performance objectives for the area 
of instruction in nonverbal communication in collaboration with 
the consultants. 

Prepare copies of goals and objectives drafted, to be given 
consultants at end of conference. 

Refine the conference draft of objectives. 

Send refined draft of objectives to consultants for further 
comment . 

Elicit comments (in writing or by phone as necessary) from 
consultants regarding refined draft of objectives. 

Write a second draft of objectives based on comments elicited 
from consultants. 

Submit second draft of objectives to consultants for examin- 
ation in preparation for a second conference. 

Elicit **clean-up” responses from consultants meeting 
as a group (2nd conference). 

On the basis of field test data, amplify and clarify objec- 
tives items. 

Add or substitute suggested content materials to use with 
objectives to include contemporary works (cf. field 
test data). 



ENABLERS: 
i UV 

1 01 Experience in the field of English. 

2 14 Experience in writing and publishing. 

2 30 An ability to be a generalist, to cross disciplines in 

coordinating sets of activities. 

3 03 Sensitive to the need of consultants to have guidance 

during the conference. 

3 29 Willingness to try many things to give direction to the 

effort and in achieving consensus . 

3 34 Blind stupidity and a lot of fortitude. 
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P-17: Response to QLssenting Position Paper (Journal Article) 



STANDARDS; 

J 

1 01 All the arguments or points selected to be answered or dealt 

with have been covered. 

1 13 Directors are satisfied with the content (responses to the 

arguments) of the paper. 

1 11 Response dealt with the arguments/points in the dissenting 

paper, i.e. , it was focused and did not ramble beyond 
issues presented in the dissenting paper. 

1 04 Response was readable and understandable by the audience 

(readers) . 

1 01 Responses/answers were complete — adequately supported 

and/or defined. 

1 07 Author of the response to the dissenting paper was 

satisfied with his paper. 

TASKS: 

m 

01 Read the dissenting paper. 

01 Read and study behaviorist theories and practices. 

01 Read most of the literature related to behavioral 

objectives. 

01 Attempt to state our (collective) philosophy as it 

stood at that particular point in time. 

01 Select the major arguments or issues expressed in 

dissenting paper. 

04 Write a "gut level" response to the dissenting paper. 

01 Determine which major points could be or should be answered-- 

deal with the substantive, philosophical views relative to 
the use of behavioral objectives in the discipline of English. 

04 Write the responsive paper in a questionning , exploratory 

style. 

04 Respond directly to those selected issues raised by the author 

to the dissenting paper. 

04 Attempt to respond in a more literary language as opposed 

to using scientific language; avoided behaviorist jargon. 

24 Review and critique draft by all project staff. 

04 Control the timing of the content of the paper. 

06 Edit paper for length before it is published. 

ENABLERS; 

5 W 

1 02 Knowledge of what a behavioral objective is. 

1 04 Knowledge of bel*aviorist theory and practices. 

1 04 Knowledge about various techniques in the behavioral sciences, 
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which may be applicable (with perhaps some modif icatiLn) 
to the field of English, which facilitate the generation 
of knowledge relative to the liumanities and the teaching 
of the humanities, 

1 02 Krowledge of the focus of a behaviorally phrased 

objective, i.e., the approach must be student centered. 

1 08 Knowledge of exactly what was to be refuted — the 

author’s arguments and not the author, 

2 35 Ability to control the tone in which the paper is written, 

2 14 Ability to write in a clear, cohesive, unified manner. 

3 06 An understanding of the adaptive, but perhaps united, use 

of behavioral objectives in the field of English, 

3 11 Sensitivity to the notion that teaching involves modifying 

behavior, 

3 30 Awareness of the audience for which the paper is written. 

3 13 Sensitivity to the effect the tone might have on the 

audience (readers) , 

3 13 Sensitivity to the effect the language in which the paper is 

written will have upon the audience. 



P-18; First Draft of Introduction to Catalog 



STANDARDS: 

J 

1 13 Critical responses received from consultants indicate 

that introductory comments about behavioral obj ectives 
are relevant. 

2 09 Suggestions coming from consultants are constructive 

relative to use of the document as an introductory section 
to the catalog. 

2 09 Constructive suggestions regarding indiviaual items of 

behavioral objectives are forthcoming from consultants. 

TASKS : 

NO 

01 Review comments and criticisms raised by consultants at 

previous conference. 

01 Read professional literature relative to behavioral objectives 

in English. 

01 Consider opposing views within the National Council of 

Teachers of English (Professional organization) regarding 
behavioral objectives . 

01 Assess (inductively) the reasons for opposition to 

specifying behavioral objectives in English. 
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04 Summarize the critical issues identified from different 

sources . 

04 Draft summary pro and con statements for each issue identi- 

fied. 

04 Prepare duplication of dissenting position paper and the 

paper written in response to it. 

04 State (write) the decision^ made in relation to questions 

raised at the previous conference about target audience, 
format, etc. 

04 Write overview of project to data. 

04 Write summary of current considerations in further refinement 

of objectives. 

04 Prepare duplicate drafts of objectives refined and/or 

generated following the previous work conference. 

29 Send drafts of paper and appropriate sections of the 

objectives specified to the consultants prior to meeting. 

ENABLERS: 

_S W 

1 06 Knowledge of the criticisms teachers direct toward be- 

havioral objectives. 

1 06 Knowledge of teacher experiences with behavioral objectives 

and the reactions to those experiences. 

1 03 Knowledge of the historical developme'.t of behavioral 

objectives as applied to English. 

2 14 Skill in conciseness and economy of words in writing. 

2 11 Ability to be direct and honest. 

3 22 Sensitive to what happens to teachers when forced to 

develop or utilize behavioral objectives. 

3 38 Sensitive to increasing teacher awareness of behavioral 

objectives and their relevance. 

3 16 Sensitive to the political climate in the various states 

and schools in relation to behavioral objectives. 

3 31 Sensitive to fairness in presenting opposing points 

of view. 

3 13 Awareness of the **purr*' and **snarl” effects of words 

in biasing a presentation. 

3 31 Sensitive to the length of a document (so as to fairly 

represent the views but not appear to beg the question). 



P-20: Outline of Catalog^ Goals, Objectives and its Organization: First Draft 

i 



STANDARDS: 



J 

1 05 



A feeling of the Project Directors that no performance ob- 
jectives are trivial. 
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1 30 Feedback from field test reflects minimum dissatisfaction 

with objective items • 

TASKS : 

m 

31 Provide time within the project to permit consultants to 

read the "pre-knowledge" ma^'erial (while in conference). 

31 Provide time for "getting the air cleared" (with consultants) 

following revif ; of background material. 

29 Obtain consensus from consultants about the nature of the 

background material and the tasks to be undertaken. 

04 Write objectives based on ideas and Inputs from consultants. 

29 Provide consultants with background mnterial on be- 

havioral objectives . 

29 Discuss with the consultants the issue of behavioral objectives 

in the humanities. 

03 Distinguish between and organize separate sections (Chapters) 

for Language and Writing performance objectives. 

01 Discuss with consultants possible translation of their 

(college and university based) material into high 
school framework. 

03 With the consultants, reduce the knowledge of linguistics 

and a translation of it into. . .an applied form 
having meaning for high school. 



ENABLERS : 

S UV 

1 18 Know the people you have asked to work. 

2 30 Have the ability to take criticism. 

3 34 Understanding that asking orofessionals for self-preparation 

prior to project sponsored activity is not likely to 
wo^k. 

3 29 Willingness to try to prove something in spite of criticism. 

3 47 Willingness to sit down and write the objectives even though 

content consultants could not do it satisfactorily. 

3 08 Possess a sense of the trivial in a content area. 

3 06 Have a view of language as a very practical thing. 

3 06 Have a view of "instruction" as a means to show people how to 

use the language. 

3 06 Have a view that the knowledge base of language provides 

a means for eliciting other desired student performances. 

3 11 A realization that much of language is gene. *ted through 

experience, e.g. , subject-verb agreement. 

3 11 A realization that the teaching of language rule by rule 

is not very successful. 




51 



5 ; 6 



P-25; List of Consultants ('Jon', irmed) | 
\ 



STANDARDS ; 

J 

2 09 (That ultimately) the project directors feel that no 

p^.rformance objectives generated are trivial . 

2 34 (That ultimately) feedback from the field test reflects 

a "minimuu' ’ of dissatisfaction with performance items. 

TASKS : 

NO 

21 Specify what you want the consultants to do. 

21 Very carefully select people to insure being able to 

get from them that which you expect to need. 

21 Assure (by checking with them) that the people selected 

can, in fact, do the job. 

21 Accommodate accessibility by selecting consultants close 

to area. (It may be a fallacy that geographic dispersion is 
prerequisite to achieving diversity of perspectives.) 

21 Anticipate diversity of consultant opinion ^ advance (.^n- 

house) and specify position pertaining thereto. 

21 Limit the selection of consultants to diversity of opinion 

on how a thing should be done rather than on whau or whether. 

ENABLERS : 

s m 

1 18 Knowledge of the people you have asked to work. 

3 48 (An attribute) of common sense, 

3 02 Understanding that "eminence in the content field" does not 

necessarily mean that the person can Jo the job (of specifying 
performance objectives) . 

3 25 Recognition of varying degrees of commitment between project 

directorate and consultants. 



P“27: Conference, Field Representative 



STANDARDS: 

J IJd 

2 34 Teachers of English in each field test site show evidence 

of having used the.icatalog. 
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2 22 



TASKS’ : 



m 

01 

01 

23 

23 

23 

23 

23 

23 

23 

23 

23 

24 



29 

29 

29 

29 

31 

31 



Teachers of English 
catalogs reflecting 



in each site return annotated 
a critique of Its content and utility. 



Determine types of schools to be selected for catalog tryout. 
Determine community sizes involved with schools (to be 
representative) . 

Obtain a complete 1-st of ES-70 schools (from OE) . 

Determine geographical locations in the U.S. to be used 
as representative . 

Select eight ES-70 schools to be approached for field test 
participation. 

Write to Administrator of each ES-70 program requesting 
cooperation in field test. 

Select eieht ES-70 schools which qualify and meet sampling 
standards set. 

As^ each ES-70 school administrator to select and elicit 
cooperation from another school in their area (noa-ES-70) 
for the field test. 

Identify and select schools local to the project institutions 
(8) to be. used in field test. 

Contact English Department heads of schools to elicit field 
test cooperation. 

Participate in meetings, phone conversations with school 
administrators to answer questions and obtain administrative 
approval for use of school as a field test site. 

Assess (unobtrusively) the conferees to identify the 
seemingly best informed, most articulate and imaginative 
to tell colleagues some of the ways which have occurred to 
them to use the catalog. 

Describe project efforts to English Department heads. 

Ask each school administrator to select a representative 
to attend an orientation conference on use of the catalog 
of performance objectives . 

Direct conferees to read the catalog and prepare questions 
to be answered. 

Specify the activities intended for thr conferees to carry 
out when they return home. 

Schedule facilities and agenda of events for field representative 
orientation conference . 

>oderate small discussion groups directed to iuentifying 
various ways teachers might use the catalog. 



ENABLERS : 

S W 

3 02 Sensitivity in selecting conferees who could offer creative 

illustrative uses of the catalog to their colleagues. 

3 16 An understanding that the leadership for the project must 

come from the professional discipline in order to obtain 
the cooperation of the professionals in the field. 
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3 lb iMe reputation which comes from having a publication on 

the teaching of high school English widely used in the U.S. 

3 16 An awareness of the need to Jegitimize the efforts by having 

a dire':torship that ipso facto understandw "tho peculiar 
problems of English.” 



P-28: Initial Work Conference (C 



I 

her, //I) ! 



STANDARDS : 

j m 

1 12 First round of behaviorally stated objectives in the different 

areas/ categories of English are generated. 

1 12 First round of general goals within each area/category of 

English are generated. 

TASKS : 

m 

01 Identify areas of English for which goals and objectives are 

to be generated. 

29 Generate sample behavioral objectives for the i> ntified areas/ 

categories of English. 

21 Generate jl tentative list of people to serve as consultants 

at this conference. 

21 Rank-order the list of consultants by preference within 

each area/categorv of Eng] Lsh. 

21 Invite (by mail) the consultants to the conference. 

23 Select conference site. 

23 Arrange with the site hotel/motel to have needed facilities 
available for the conference. 

31 Draft the agenda for the conference. 

24 Send draft to all the Directors for their review and critique. 

29 Sent materials in advance of the conference to all of the con- 

sultants, e.g., samples of behaviorally stated objectives, 
Mager*s "Preparing Instructional Objectives," an abstract of 
the project’s proposal, etc. 

01 Determine conference objectives or expectations. 

31 Design the arrangement for small group work — determine which 

consultants will be grouped together and how they are to 
be grouped. 

31 Reconvene as a group several times during the conference to 

discuss questions which are common among the various small 
groups ^ 

31 Work primarily in small groups (within an area/ category 

of English) and individually to generate behaviorally stated 
objectives. 

54 

95 






f 



25 Provide a couple of social gatherings during the conference. 

25 Meet (directorj) on the second afternoon of the conference to 

**take stock** of the conference in general. 



ENABLERS ; 

S W 

1 06 Know who the recognized authorities are in various areas 

within the field of English. 

1 09 Know how to make arrangements for and at the conference site. 

3 01 Sensitive to how authorities regard themselves — 

sensitive to their self concept . 

3 02 Sens:ftive to how easily each consultant is able to work with 

other people. 

3 02 Sensitive to the general attention or work span of most people 

in planning a conference some allowance must be made for 
breaks in work session. 



1 

I 
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Chapter I: Overview 



This ovcp/lew presents a brief synopsis of the Instructional. Ob- 
jectives Exchange (lOX) Project as an Introduction. This Is elaborated 
by a discussion of the objectives^ rstlonalei and significance of the 
project and the context In which It operates. 



Synopsis of the Project 

Title: Providing Wide Ranging i Diversely Organized Pools of 

Instructional Objectives and Measures. 

Responsible Institution: University of California at Los Angles, 

Instructional Objectives Exchange. 

Funding Source: U.S. Office of Education, National Center for 

Educational Research and Development. 

Funding Duration: July 1969 to June 1971. (24 months) 

Observation Date: April 1971. 

Present Stage of Development: Final stage. 



RDD&E Focus of Project: Educational development. 



Expected 

Outcomes: 



1. New catalogs of instructional obJe:tlves and 
measures. 

2. Quality control procedures for such objectives 
measures. 

3. More functional r ■'cegorlzatlon schemes for 
objectives and measures. 



Level of Funding and Duration: Medium-Low. (level 3 of 7 levels) 



Agency Setting: University. 

Setting of Primary Location of Work Effort: Instructional 0./Jectlve^ 

Exchange offices. 




Staff Summary: Three-man directorate. 

6 - Research Associates. 

1 - Clerical support staff. 

Full Time Equivalency (FTE) : On this project, assigning a meaningful 

FTE for personnel is inappropriate because of the way in which 
staff combine lOX activities with other assignments, and because 
of the additional support of lOX aside from this project, which 
has now been established as a nonprofit corporation. 

iCj. 
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Obiectlvcs. Rational*, and Slaalflcnn e of the Project 

Obi actives . The focal outputs ennunerated In the synopsis are Inter- 
secting rather than mutually Independent. The first Is the main tangible 
output of the project, to produce Improved and more comprehensive objectives 
and measures of Increased utility to ES *70 and other schools. The purpose 
of the second Is to establish quality control to assure the operational 
validity of the objectives and measures generated, and to provide progres- 
sive Improvement In them on the basis of this control. The third Is to de- 
velop classification and synthesizing schemes for Instructional objectives 
which will depart from those currently employed, thereby enabling the objec- 
tives to achieve more functional and precise operability. 

Rationale . The original coimltment of the Exchange was to solicit and 
collect objectives and measures, and to collate them Into collections by 
subject area and grade level. This was based on the thinking that In 
today's harried, public school environment, teachers rarely have time 
to become generators of their own operationally stated objectives and 
the measures that go with them. However, It was believed that much good 
work was being done In relatively Isolated Instances. By collecting and 
sharing the outputs from this work, teachers would be able to select 
needed objectives, rather than having to generate their own, and thus 
derive the benefit of work In the field without unnecessary duplication 
of effort. 

The original solicitation effort was less successful than anticipated, 
although the demand for the outputs thus collected was large and continues to 
grow. Examination of the materials collected showed Inconsistencies that 
led In part to the generation of the objectives of this present project. 

Somi^ objectives collected contained ambiguities, or In other wayj lacked 
operational validity, and had to be screened out or modified to eliminate 
or reduce the defects. Also, while some subject areas and levels offered 
such an accumulation of objectives that the use of all of them would 
result In redundancy, there were other areas where there were gaps. 

Objective one, therefore. Is to take Immediate steps (during the life 
of this project) to fill In the deficiencies and gaps that became evident 
from pursuit of the simple "exchange" notion. The Exchange was started 
as a "sharing" center and. In spite of these technical deficiencies, has 
proved successful In providing a source of objectives and measures not 
available before. An ever-increasing demand has continued to develop for 
the objectives provided by the Exchange. 

The second objective In a sense pursues current concepts of objectives, 
providing a framework within which continued Improvement may be generated 
by quality control. The major thrust of this effort is presently limited 
to established fields, such as reading, science, fundamentals of English 
usage, mathematlcc, social studies, foreign languages, and vocational 
specialties. 

The third objective Is based on a longer-range rationale. Current 
objectives and measures are classified by conventional subject and grade 
level and by topic within these areas for division or categorization. 

Several studies have progressively suggested that these conventional areas 
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for specification and measurement are not truly indicative of the learn- 
inK tliat occurs in the individual student, or of the ef fectlveuess of 
instruction in producing such learning across a collection of students 
in a class. For this reason, the third objective is to examine the 
results of work in this area and to take first steps toward the develop- 
ment of new classification schema tliat may be more merniiigful than the 
traditional variety. 

Context in which the Project Operates 

This project is being pursued by a close-knit team which appears 
to work well together and to Interact in a way such that initial dif- 
ferences of input, due to difference in background, experience, or former 
conditioning, brings about reinforcement: a stronger end result than 

would be possible from a "sum of the parts." The work of producing 
the project's outputs is performed by individual members of the team, or 
small (two- or three-man) combinations of them. 

If "project" is identified with the people who perform it, then 
"context in which it operates" would appear to refer to the physical 
surroundings in which they work as a team. This Identification is some- 
what Inadequate to describe the actual context of this project. Pursuing 
this Identification, the "context" would be the offices of the Instructional 
Objectives Exchange located just off the UCLA campus. This, it is true, 
is where personnel of this project were interviewed. But a more accurate 
description of the project context would be to say that the work goes on 
in project members' heads, wherever they nay be at any given moment, that 
outputs take on tangible form by being committed to paper, and that the 
needs of team operation are met by their getting together from time to 
time at the above-mentioned offices. 

Perhaps this viewpoint of context also links with the problem of 
specifying a precise staff FIE, although there are other reasons for 
that. The notion of a specified FTE does connote individuals allocating 
a certain amount of their time in well-defined "slots" of activity. This 
describes the way the real world operates on this project. 

The parent agency, in some respects, can be considered to be the 
Instructional Objectives Exchange, which was Initiated at the UCLA Center 
for the Study of Evaluation. This project was funded by the U.S. Office 
of Education (USOE) as a project of the Instructional Objectives Exchange, 
at the time part of the Graduate School of Education at UCLA. When the 
number of collections of objectives began to become ready for dissemina- 
tion, the question of how they could best be made available to a larger 
audience was considered. The advisors at the Center considered this ac- 
tivity to be more a service than a research or development activity. For 
this reason, in April 1970, lOX (Instructional Objectives Exchange) moved 
out of the Center and became a nonprofit corporation. 

The profit from the sales of the objective collections continued 
to be returned to the Center until July 1970, when a continuation of the 
pr< jct for its second year was granted. After that time, profits from 
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the sale of the objective collections were returned to T'V . Funding 
for the project continued to be administered through the Graduate School 
of Education. In addition to these project funds the ne^/ly-lncorporated 
lOX began receiving funds from USOE Title 111 monies through 18 states. 
This came as a result of a solicited proposal to generate more collec- 
tions of objectives, particularly In the attltudlnal domain. The Bureau 
of Indian Affairs (BIA) also provided funds following a solicited proposal 
of parallel nature. 

Figure 1 shows a somewhat simplified arrangement of these relation- 
ships. A further useful relationship for the project Involves the teach- 
ing assignments of two of the 3-man directorate. Tliey teach In the Pro- 
duct Research Training Program. Students In this program receive training 
that Is also useful for the generation of objectives. These students pro- 
vide a reservoir of trained personnel that has been tapped by the project 
and the operational lOX throughout Its history. 

The physical environment for the project s"aff Is presently In the 
lOX offices which have recently been expanded to provide ample work 
space and more privacy. Support facilities and libraries are readily 
available both within the University (easy walking distance) and the City 
of Los Angeles. The order and shipping warehouse (not part of this pro- 
ject, but part of the operation of the nonprofit corporation that sells 
objective collections) is located in San Fernando, only minutes away from 
lOX offices by freeway. 
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Chapter II: Parameters of the Project 



This chapter discuHSes the staffing pattern of the lOX Project, 
Includes a roster of staff, describes the outputs being generated, 
and shows the dependent relationships of the outputs in an output 
map • 



P r oj e c t S t r u c t ur c 

Staf f structure « Figure 2 graphically Illustrates the staffing of 
this project. Along with the Project Director two other lOX 
stiiff members make up the 3-tnan directorate. Major decisions within 
the project are made by majority decision of this 3-man directorate. 

Under the directorate, six staff members designated as research 
Associates are responsible for generating the outputs described In 
tills profile. In addition to the people who produce the project's 
outputs, there is a clerical staff of one and a shipping clerk. The clerk's 
responsibility is to ship out Exchange products (which are ultimately 
derived from the projects output) in response to orders received from 
the field. This latter activity is not conducted at the lOX offices, 
but at another location. 

The project is unique in that for all except the clerical help 
the staff are largely supported by other activities, although they 
may spend a major part of their time working for this project. 

Project roster . The following is a roster of project personnel. 

Those people interviewed are identified. 

1. Project Director * He is responsible for interfacing with 
other agencies, and is chairman of the 3-man directorate which makes 
major decisions regarding the objectives and measures to be generated, 
structure, format, etc. (interviewed) 

2. Directorate . Three people, including the Project Director, 
who jointly preside over decisions to be made In the course of the 
project . (all interviewed) 

3. Research Associates . These people (presently six in number) 
are responsible for generating the objectives and measures and putting 
them into collections. (not interviewed) 

4. Clerical Staff . This staff responds to phone calls, attends 
to incoming correspondence, handles routine typing, etc. (not inter- 
viewed) 
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FIG. 2. Staff structure. 
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O utputs Generated 

Figure 3 shows how the library of collections that constitutes the 
focal output of this project is built. In addition to the objectives 
and measures that make up the collection, there is a commitment t? 
provide guidance in their use. The intent is to make this as open as 
possible. This is clearly stated in the publications of the project. 

Figure 4 shows a generic set of subproducts, some or all of which 
may appear in generating each tryout version. In the concern for pro- 
ducing, e.g., the final published collection; and again when collections 
are revised for republication at some later date, the project exercises 
a quality control influence by the mechanism shown in Figure 5. This 
varies according to the collection to which it is applied. Figure 6 
shows a generic set of subproducr.s that go into producing the data summary 
responsible for providing this quality control. In a sense, ligures 3 and 5 
represent different **dimensions^' in the relationship between outputs that 
intersect, as will become evident as the description proceeds. 

Looking further ahead. Figure 7 shows the production plan for generat- 
ing a revised classification scheme, with a view to providing morf useful 
objectives and measures as tools. The present series are felt to be 
inadequate by virtue of the constraints imposed by arrangement within 
time-honored discipline boundaries. 

Index of products . The products identified in Figures 3 through 7 
are listed following these figures. The first three products are major 
headings for classes of products that correspond with the three stated 
objectives of the project. Each of items P-04^ through P-11 then commence 
with a tryout version, generically designed P-13. Also shown in Figure 3 
are the development stages of item P-12 using item P-42 through P-44. 

Figure 4 chows the generic structure of P-13, using items P-14 
through P-20, some or all of which may be relevant to any individual 
final outcome item in the groups P-04 through P-11. 

In Figure 5, item P-02 is subdivided Into parts represented by 
items P-21 through P-23, according to the point along the development 
of an individual collection at which the control is being exerted, and 
these further subdivide into items P-24 through P-27, the Ifist of which 
derives from P-28, and all of which are dependent on a data summary 
generically designated P-29. 

Figure 6 gives the generic basis for each c ta summary, including 
items P-30 through P-34. 



^Each output is identified by an arbitrary number consisting of two 
parts: (a) a letter P, E, or C (product, event, or condition), and 
(_b) a sequence number irrespective of the letter. 
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P-13, Tryout Version of Collection). 



FIG. 5. Output map (generic structure of P-02, Project Quality Control Schema). 
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PROJECT QUALITY CONTROL SCHEMA 
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FIG. 6. Output map (generic structure of P-29, Data Summary). 



FIG. 7. Output map (generic structure of P-03, Classification Schema). 
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Figure 7 takes the third major intended output of this project, 

P-03 and sets forth a projected structuring of the products leading 
to its generation, including items P-35 through P-41. 

In the interviewing of personnel at this project , specific 
intersections on these items were interrogated around. These form 
the final portion of the listing below, following P-44. For the 
purpose of identifying the data as assembled for analysis by the 
computer, these items are identified by the intersections of categories 
shown on Figures 3 through 6,. 

P-01. L ibrary of Collections Containing Behavioral Objectives 
& Measures . This is the major output of the project. 

It is thus continually in progress, in the sense that 
new items are being worked on to add to it, while others 
may be being updated or improved. 

Quality Control Schema for the Library . This is the 
second major commitment ot the project. As such, de- 
velopment of improved methods for ensuring quality 
upgrading are continually being developed. 

P'~03. New Classification Schama for Behavioral Objectives. 

This is the third ma^jor commitment ot the project. 

Actual tangible products in its development are still to 
be developed as of the time the project was visited. 

The responses that form the basis for the information 
were those projected as steps in development 
of a suitable schema. (This item was interviewed 
around. ) 

P-04. C ollection of Sociology Objectives and Measures . This 

is one of the new items being worked on, as of the 
time of the site visit. 

P-05. Collection of Anthropology Objectives and Measures . 

Another new collection being developed. 

P~06. Collection of Government Objectives and Measures. 

Another new collection being developed. 

P-07. Collection of Attitude (Tolerance) Objectives and Mea- 
sures . A relatively new collection, only recently finished. 

P""08. Collection of Attitude (Judgement) Objectives and Mea- 

sures . A new set of objectives being developed as a 
result of a need perceived by staff who worked on P-07. 
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P-09. 

P-10. 

P-11. 

P-12. 

P-13. 

P-14. 

P-15, 

P-16. 

P-17. 

P-18. 



Set of Collections of Cognitive Qblectives and Measures . 

These collections have been in existence for some time, 
most of them since the time when the Exchange collected 
existing objectives and measures. 

Set of Collections of Psychomotor Objectives and Measures . 
Another part of the existing set, distinguished by the 
different type of objectives included. 

Set of Collections of Attitude Objectives and Measures . 

An effective group of measures within the existing set. 

Revised Usa^e Manual Explaining Some Potential Uses of the 
IPX Objectives . A still to be produced manual for which 
the ground-work has already been laid . 

Set of Field Tryout Versions of the Collections^ One for 
Each Collection . For any given collection this may present'* y 
be completed, worked on, or yet to be started. Thus, in the 
generic sense, production of tryout versions is an ongoing 
activity . 

Set of Inhouse (Bench Test) Versions of the Objectives and 
Measures (at least one for each tryout) . This precedes the 
tr>^out version (P-13) and is a sort of bench-test set, tried 
out under quite close supervision, prior to being released in 
an adequate tryout form. 

Set of Introductory Rationale Statements for Some of the 
Collections . When the objectives are completed in a form 
for publication, these will foirm an introduction to the pub- 
lication containing them. However, they may be an expressed 
(written or otherwise) statement prepared before the objec- 
tives, and thus serve as a basis for such preparation, or they 
may be generated merely as a guide to the user. 

Set of Objectives Statements Collected from External Sources . 
While objectives which were the major source for items P-09 
through P-11 were submitted from outside sources, the trend 
has been to develop most objectives Inhouse (item P-18). 

Set of Test Items Collected from External Sources . When 
objectives were collected from external sources, some test 
items came with them. Also, in some Instances, additional 
items needed generating to round out the collection. 

Objectives Statements Generated Within IPX Project . These 
were originally generated for the purpose of completing col- 
lections where items gathered from external sources had identi- 
fiable gaps. More recently, virtually all objectives for a 
collection are generated within the project. 
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P-19. Set of Test Items Generated Within IPX Project. Originally, 
these were merely to round out sets collected from external 
sources or to provide test items where they were lacking. 

The current trend is to generate all test items for a collec- 
tion within the project. 

P-20. Technical Paper //3 . A paper produced earlier in the project 
and still used serves as a guide for new people on the staff 
in generating objectives and measures. (interviewed around) 

P-21. Quality Control Schema for Tryout Versions of Collections . 

This is an ongoing, developing means of upgrading quality at 
the most sensitive stage of development. (Both subdivisions 
of this item were interrogated around.) 

P-22. Quality Control Schema for Sales Versions of Collections . 

Application of data, largely obtained during the tryout phase, 
to ensure highest quality of sales version. 

P-23. Quality Control Schema for Revised Versions of Collections . 

Utilizing longer term feedback to improve quality used when 
it is time to bring out a revised version. 

P-24. Quality Control Schema for Objectives (Primarily Cognitive 
Collections) . Procedures related to objective measures. 

P-25. Quality Control Schema for Measures (Primarily Affective 
Measures) . Procedures related to the affective measures. 

P-26. Quality Control Schema for Preferential Data on Objectives , 
(interviewed around) 

P-27. Quality Control Schema for Screening of Measures for Revision 
(Primarily Affective) . 

P-28. Report on Quality Control Screening Program . 

P-29. Data Summary for Quality Control Program . This is the 
generic output which may be in present, past, or future 
item descriptions. 

P-30. List of Data Sources for Quality Control Program . Where a 

number of sources are used for quality contro. data, this will 
serve as a checklist for their utilization. 

P-31. Data Collection Procedures for Quality Control Program . 

P-32. Data Collection Instruments for Quality Control Program . 
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P-33. 


Data Analysis Procedures for Quality Control Program. 


P-34. 


Criteria for Quality Control Program, 


P-35. 


Reclassification of Existing Collections . This is presently 
seen as a sheet to be included in existing collections. It 
will indicate th.» reclassification of objectives and/or 
measures contained in that collection under the new 
schema. 


P-36. 


Physically Classified Collections Under New Classification 
Schema. This will be a rounded out collection, built 
according to the new richema, with categories filled as 
revealed by the new schema. 


P-37. 


New and Revised Objectives and Measures. Built according 
to the new schema. 


P-38. 


Procedural Steps for Sorting the Existing Objectives in 
Collection. Devised to suit the new classification sets. 


P-39. 


Procedural Steps for Revising Existing Objectives into 
New Classification Sets. 


P-40. 


Tentative New Pragmatic Classification Schema. A working 
design to initiate anew arrangement of categories of ob- 
jectives. The objectives will be further refined on the 
basis of the experience that will be generated by the pro- 
cess of classification . 


P-41. 


Review of Proposed Classification Suggestions. A review of 
all proposals in the literature having some promise of pro- 
viding viable new classification schema (interviewed around) 


P-42. 


Summary of Data on Actual Use of Collections in the Field. 


P-43. 


Data on Actual Use of Collections in the Field. 


P-44. 


Presently Used Pamphlet on Potential Uses of lOX Objectives. 



The following items have been given identifying P numbers, but each 
of them is an intersection of two sets of generic categories in the foregoing 
section of the list. The identifying intersecting product numbers follow the 
individual P number. Thus P~06/P~15 signifies an intersection of P-15, an 
introductory rationale, as an element of P-06, the collection of attitude 
objectives. All of the following items were interviewed around, except where 
otherwise noted. 



P-45. 


P-06/P-15. Rationale Statement for American Government 
Objectives . 
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P-46. 

P-47. 

P-48. 

P-49. 

P-50. 

P-51. 

P-52. 

P-53. 

P-54. 

P-55. 

P-56. 

P-57. 

P-58. 

P-59. 

P-60. 

P-61. 



P-08/P-13. Tryout Version of Cognlclve Judftment Measures . 

In the course of generation, some tryou-s had been given. 

The finalized tryout version was still to be produced at 
time of visit. 

P-08/P-14 . Inhouse Version of Cognitive Judgment Measures . 

This existed at time of visit and had been used for tryouts. 

P-08/P-15. Rationale Statement for Cognitive Judgment Measures . 
This statement sparked the activity to generate this collection 
of measures . 

P-08/P-18. Staff Generated Objective Statements for Cognitive 
Judgment Measures . These are statements of the objectives to be 
measured based on the rationale and literature search. 

P-08/P-19. Staff Generated Test Items for Cognitive Judgment 
Measures . Items designed to measure the objectives generated. 

P-12/P-29. Summary of Data on Usage of Objectives and Measures 
for Usage Pamphlet . This product is in the process of assembly, 

P-21/P-24 . Quality Concrol Schema for Tryout Version of 
Cognitive Objectives . 

P-21/P-25 . Quality Control Schema for Tryout Version 
of Affective Measures . 

P-04/P-14 . Inhouse Vt*rsion of Sociology Objective Collection . 

P-27/P-29. Data Summary for Affective Measures Quality 
Control . (Item analysis) 

P-27/P-33. Data Analysis Procedures for Affective Measures 
Quality Control . 

P-26/P-29. Data Summary for Cognitive Objectives Quality 
Control . (Item Analysis) 

P-26/P-30 . Data Sources for Cognitive Objectives Quality 
Control . 

P-26/P-31. Data-gathering Procedures for Cognitive Objectives 
Quality Control . 

P-26/P-33 . Data Compilation Instrument for Cognitive Objectives 
Quality Control. 

P-26/ P-3 3 . Data Analysis Procedures for Cognitive Objectives 
Quality Control . 
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P-62. 


P-07/P-14. Inhouse Version of Attitude Objectives Collection. 


P-63. 


P-22/P-24. Quality Control Schema for Sales Version of 
CoKnitlve Objectives. 


P-64. 


P-23/P-34. Criteria for Revising Sales Versions of Objectives 
and Measures. (not Interviewed around) 


P-65. 


P-22/P-25. Quality Control Schema for Sales Version of 
Affective Measures. (not Interviewed around) 


P-66. 


P-27/P-28. Rough Draft of Data Report for Revised Sales 
Version Attltudlnal Measure Quality Control. 


Events and conditions of the proiect. At this prolect an extremely 


Informal, helpful, and cooperative atmosphere prevails. Management takes 
the form of a 3-man directorate to whom all major decisions are referred. 
Being located In convenient proximity to UCLA means that personnel have 
relatively easy access to subject matter experts for advice whenever needed 



Thus» as revealed in the detailed responses in Appendix A, the staff refers 
to appropriate faculty or other campus resources on an Informal basis when- 
ever the occasion demands or, even lightly, "suggests." 
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Chapter III; Summary of the Data 



hacli of the tnhlG« included in this chanter summarises a cateporv of 
data obtained in relation to the various outputs identified for the lOX 
Project. The column labeled **ProJect Outputs** identifies the outputs of 
the project which are appropriate to that table. (Only those outputs for 
which data were obtained during the interviews with staff are included.) 

The categories of data shown in the tables are the standards by which the 
satisfactoiy completion of the outputs are Judged, the tasks required to 
generate an output meeting those standarde; and the enablers (knowledges, 
skills, sensitivities) which facilitate the carrying out of those tasks. 
Within each of these categories is found a list of descriptive labels 
which are representative of interviewee statements. In the process of 
reducing this interview data, narrative interviewee statements were first 
linked to one of the above categories, then classified by means of a num- 
ber code under the most representative descriptive label. Each table pro- 
vides the frequency with which interviewees cited specific statements (rep- 
resented by the descriptive labels) of each category. ^ 

Standards have been assembled into two category sets: Those based 

on management considerations, and those strictly related to the product 
or output against which they appear. Table 1 shows the management 
(process/operations) standards, listed in columns with their set desig- 
nators (LM) . In the Appendix listings for standards, these are associated 
with the digit **2** in the J column. Thus, against Output P-41, a management 
standard with the number 36 appears in Table 1, which bears the category set 
label, **36 - employment criteria met.** In the Appendix, under Output P-41, 
the first standard, with designator 2 36, reads, **people selected were cap- 
able of generating alternative hypothesis,** which indicates in what respect 
candidate(s) (plural in this case) met criteria. 



If the reader is interested in the narrative statements of the 
interviewees, these can be found in the Appendix. To locate the 
narrative statement for any given category, first note the output 
and its identification number in the table. Second, note that each 
descriptive label within a given category has a distinct number or 
code. , Turn to the Appendix and locate the output. Under the output 
locate the category label or heading (standard, task, or enabler) and 
pinpoint the number or numbers (depending on frequency cited) of the 
descriptive label which appeared in the table. The statement in the 
Appendix opposite this number is the original narrative statement from 
an interviewee and is only represented in the table by the descriptive 
label and its number coding. 
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Output P-66 presents an interesting example of the value of the 
verbal statements contained in the Appendix, because both management- 
standards (this refers to the type, not necessarily to the relevant 
judgments being made by management) bear the same designator: "14 - 

no felt deficiencies." The wording of the entries numbered 2 14 under 
Output P-66 in the Appendix show*^ that these were actually different 
bases for making that kind of juiigirtcac. 

Table 2 Ijsts the production standards. These standards were much 
more plentiful on this project. interer.ting set is the one designated 
"01 completeness of content." Aga.^nst Output P-03, "A New Classification 
Schema," appears the description, ’the system includes objectives ap- 
plicable across subject fields." Against Output P-54, Generated Objen- 
tives/Test Items for the Sociology Collection, appears the description, 
"feel field is covered in each area as additional review of literature 
fails to turn up new areas." And against Ch\tput P-66, "Rough Draft of 
a P'lta Report-* ■," testing design included all grade levels and some 
variety in socio-economic backgrounds." Completeness of content, there- 
fore, has three different contexts within this one project. Other stan- 
dards can present equally interesting studies. 

Table 3 lists the tasks associated with each output analyzed. The 
remaining three tables, 4 through 6, tabulate tne knowledges, skills, 
and sensitivities educed from the interviews relative to the outputs 
•.ney benefit. 

In Table 4, the one variety of knowledge that spreads fairly 
uniformly across the production of all outputs has a total frequency 
count of 13 (out of 48 total across 15 categories of knowledge). It 
is knowledge of RDD&E subjects. Examination of the words used In 
the appendix, against entries for 1 03 in the enablers listings for 
Outputs P-20, P-49j P-53, P-55, P-56, P-57, P-60, P-61, and P-66, 
show that the total loiowledge required covert: quite a range. 

In Table 5, the skills listed are somewhat scattered, with a 
total frequency count of only 29. The biggest concentration of skills 
was obtained in response to interviewing around the data summary for 
cognitive objectives quality control (P-57). 

In Table 6, the distribution of sensitivities also looks like 
a scatter diagram, with the exception of one variety of sensitivity 
that is spread fairly uniformly across the set: 3 02, described 

generically as sensitivity to capabilities and llmltatious. However, 
the overall conclusion, by comparing the three tables. Is that personal 
enablers having the general characteristic's of knowledges and sensitivi- 
ties are each efted about twice as often as those that may be considered 
as skills . 
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Chapter IV: Supplementary Data 



This chapter contains information about the backgrounds of the staff, 
the resources and equipment available for use by the staff in carrying out 
project tasks, the management of the project, and the classifications of 
output characteristics. 



Summary of Staff Background 

Of the eight professional people interviewed, the three that constitute 
the decision-making management group each had doctorate degrees, the re- 
mainder baccalaureates. Project salaries may not reflect a true picture 
of remuneration for work on this project, because each of these personnel 
has other commitments and teaching assignments with which they share their 
time. 



Academic majors concentrated in education/ teaching, with some support 
in psychology and in statis tics /measurement. The three decision-making mem- 
bers of the team each had published more than 16 professional publications 
(the highest figure on the questionnaire for which a checkmark was provided). 
They aggregated 42 years of experience teaching in a college or university 
setting, 23 years of working in education conducting research and teaching 
in a school setting, four years of working in educational R & D centers, and 
various other experiences such as working with the Peace Corps. 

Within the above experience is a total of 32 years directing educa- 
tional research, development, and evaluation efforts. 

In response to questions about the level of education necessary for 
the job.^ those with doctorates each answered that a lower degree would 
be appropriate. Of those with bachelor's degrees, two responded that 
the job required a master's degree. 

As stated elsewhere in thic profile, the working atmosphere on 
this project was enthusiastic — they believe quite strongly in what 
they are doing. Because of this, a work sharing takes place, based on 
a combination of personal Interest, and of seeing "what needs doing" 
that serves to distribute the load for optimal progress. 

In the job descriptions most of the professional staff list them- 
selves as "research assistants." While there are various parts of the 
overall work to do, l.e., writing objectives and critiquing them, writing 
measures and critiquing them, collecting and processing data from the 
field to determine various aspects of quality control, etc. These jobs 
do not fall into the clearly defined patterns they might have assumed on 
a more formally organized project. 

Each staff member has areas of interest in which he takes a particular 
interest and does most of the work. In addition to this, each responds to 
work loads of other staff members as the overall pressures indicate the 
need. 
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Support Resources 

The following items were listed by various project staff members as 
being available from other persons or agencies for use in this project: 



Printing. 

Mimeograph . 

Xerox. 

Technical writing. 

Editing. 

Secretarial service, (other than typing) 

Typing. 

Purchase of supplies and equipment. 

Library holdings. 

Subscriptions to technical and professional journals/periodicals. 

Requests for documents or publications not locally available. 

Computer analysis services, (data processing) 

Computer program writing. 

Statistical consultation . 

Subjects for experimentation or tryout of procedures. 

Travel arrangements . 

Budgetary and other fiscal accounting. 

Scoring of test items. 

Equipment listed as immediately available and used on the project, 
other than typewriters were: 

Dictating equipment. 

Desk calculators. 

Most of the professional staff stated that they made trips for the 
project, some from 3 to 5 a year, others up to more than 25 a year. Trips 
of longer distance would be of 2-day duration, shorter trips of less 
than 1-day duration . 

Project personnel also reported that a reasonable amount of 
time was made available to them for preparation of professional papers 
for publication, presentation of speeches at conventions and professional 
meetings, personal reading of current professional literature, estab- 
lishment and maintaining of contact with other agencies and potential 
sponsors, and interaction with personnel on other projects within the 
organization. 

The personnel also reported that they were expected to prepare 
professional papers and make presentations, as outlined in the previous 
paragraph. 



Summary of Project Management 



The questionnaire also provided the respondent (interviewee) the opportunity 
to describe the project management. The Project Director and his staff 
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see the management very similarly, with slight individual differences 
of expression. The Project Manager and most of the staff do not see 
any specification of detailed time lines, although a few of the staff 
differed on this. Virtually unanimous were statements in support 
of weekly or biweekly staff meetings to review project progress, and 
about annual or semiannual review of individual perfonnance and 
effectiveness. 

On the subject of project structure, as viewed by project person- 
nel, the questionnaire listed with a brief descriptor of each: pyramid, 

corporate structure, inverted pyramid, chain of command, leader less, 
and other. Curiously some members of the staff checked each of these 
as describing this project. Perhaps the one that checked *’other,** 
with the following explanation, "We’re not leader less, but more of an 
informal cooperative team," provides the explanation for the other 
variations. That certainly describes the visiting group’s perceptions 
of the project as a functioning structure. 

In other projects visited interviews with the Project Director 
often elicited data about a condition called "project staff morale," 
in which the Project Director described tasks, enablers, and standards 
to indicate that satisfactory staff morale was achieved on the project. 

It would seem significant that that subject did not receive attention 
at this project. 

The visit was initiated with a meeting at which the interviewing 
team met the entire project staff. As the visiting team explained 
and were questioned about the purpose of the visit and the kinds of 
information sought, the response was so open that, if the visiting 
team thought of morale at all, it was with the thought that this 
project did not have a problem in that area. 

Classifications of Output Characteristics 

Outputs may be categorized in terms of a number of variables. Among 
these are (^) Structure (products, events, or conditions), (^) Function 
(policy setting, management, or production), (c) Level (focal, component, 
or facilitating) , (jd) Character (knowledge, technology, implementation, or 
information), and (^) Stage of completion. These five schema are repre- 
sented in Table 7 for each project output identified, with frequencies sum- 
marized for each category. 
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Chapter V: Project Dynam Lcs 



Project Focus 

When the visiting team was being briefed for this project, the 
name of the project location, '‘Instructional 01)jectives Exchange," 
led them to the immediate conclusion that this project's primary 
focus must be diffusion. But, while the work lixamined during 
Che visit is conducted within the agency that bears that title, the 
specific proposal under which the woik is done does not address dif- 
fusion per se. 

The dispersal of objectives produced and published is carried out 
from a separate location that the team did not even visit. The project 
receives reports of sales from this facility as well as 'funds. For 
this reason, the main concern of the proiect staff is not to achieve 
diffusion of the products of their labors. Their concern 
is with the various aspects of producing products for such diffusion. 

There is little question that the primary focus of the 
effort, as well as of the contract, is upon development of quality 
objectives. The project proposal devotes some space, under purpose 
1, to a discussion of the problem of "proselyting." It should be explained 
that the word applied to objectives, not people, and the proselyting 
referred to is the persuading of people who already have some objectives 
and measures of acceptable quality, to surrender them to lOXi The 
problem seen in the proposal and described by the word "proselyting" 
was that the staff needed to go out and find teachers willing to part 
with objectives and measures for use by the Exchange. 

The problem of proselyting apparently Involved even more than per- 
suasion. Early proselyting efforts revealed virtual nonexistence of any systematic 
collections. As a result, it became expedient for the project 
staff to turn cheir energies toward generating quality objectives 
and measures, co fill the gaps discovered in the collections they 
already had. 

This change of emphasis made the approach to purposes 2 and 3 
of the proposal much easier. Exercising quality control and de- 
veloping ne^" classification schema, while working principally with 
ccllectionn assembled from outside sources, posed problems typical 
of such activities associated with heterogeneous collections . When 
does one change, when does one rewrite, and so forth? Since the 
original commitment was primarily to circulate what is, while 
exerting some quality control over what gets circulated, there 
was an implicit commitment to retain some identity within the ob- 
jectives and measures collected and circulated. 

The transition toward total internal generation of objectives 
released the operation from this implicit commitment! there is 
no question of "murdering" somebody else's brainchild. 
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At the beginning of the time line of this project, which covered 
two >ear8, from July 1969 to June 1971, lOX operated within the UCLA 
context as an activity supported by the Graduate School of Education 
and the Center for the Study of Evaluation. During the second year 
the distribution of the collections was put on a private, nonprofit 
basis and the operation removed from the UCLi\ campus. 

However, the development work in this project, under the proposal 
submitted at the beginning, continued to be funded by USOE. As stated 
earlier, the actual mailing of packages is handled at a different loca- 
tion from the one visited in connection with this project. 

While the development is, in a sense, inseparable from the distribution 
or diffusion (which was the reason why we intuitively expected this project 
to **look like** diffusion), the activity reported on is intensely concerned 
with development. The diffusion that follows is seen as a measure of 
the quality or success of this development activity. 

Because of this almost mushroom-like development — the rapidly in- 
creasing demand for the products of this project — people have tended 
to do what is needed in response to such demand. In addition to following 
their own concerns about the direction of the project, an example of this 
response is the development of the cognitive judgment collection to comple- 
ment the tolerance attitude collection. 

Few of the personnel, except secretarial staff, work full time on 
this project. In fact some who work on it have zero assigned FTE. They 
are supported by the Exchange. This is a sort of **f8ct cl life**, 
largely due to the success of the project, (using that word here to 
cover the Exchange as a whole, rather than merely this development 
aspect) that enables successful expansion to proceed. 



Nature of the Produce 



A few words are needed to explain what the people on this project see 
as the nature and function of instructional objectives and in particular of 
the collections they are developing. 

Use of the word **objectives, ** they feel, is apt to convey to many 
uninitiated persons an implication cf **behavioral,** along with whatever 
implications the combined term "behavioral objectives" may have already 
acquired for such individuals. Sensitivity to this resistance has been 
heightened by overt publicity by certain groups, claiming that **behavioral 
objectives*' is a euphemistic term to conceal "brainwashing." Assertions 
of this nature have been made to project staff on campus, although the 
asserters admitted never having encountered an "objective." 
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The concept behind the project's use of the words instructional 
objectives" is that the measures may be applied not only to measure the 
behavior of individual students, but to measure the quality of instruction 
as well. Effecting desired changes in student behavior through instruction 
reflects the occurrence of learning. To fulfill this specific purpose, 
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dtudent responses would carry no names and results would be collated to 
determine the aggregate effect upon a whole class, rather than the effect 
on individual students. 

While that is a rather specific implication within the words *’in- 
structional objectives,** it need not be n enforced constraint upon the 
way these collections; of objectives can be used. In fact, once a school 
or person hau purchased a collection of objectives, how it gets used 
is no longer under the control of the agency that constructed the state- 
ments and measures. Even if precise instructions were included as part 
of the test instruments in order to restrict them to specific uses, it is 
a relatively simple matter to change these and reproduce the same instru- 
ment in a different context. 

For this reason, the approach adopted by the Exchange and this 
project is not to specify an exclusive manner of use. Rather, in the 
paper describing potential uses, a number of possibilities are described. 
In the booklet that will eventually replace this paper as a more com- 
plete use guide more information about possible uses, expected results, 
and methods of evaluating responses according to the intent of the mea- 
surement will be included. These changes will be based on data now being 
collected and collated. 

The project staff are finding that these diversified possibilities 
for use are aiding considerably in the acceptance of the product. Various 
people have resisted the use of objectives, of whatever precise type, 
.largely through ignorance about what **objectives** are. Possibly a *'brain- 
washing** connotation still clings. 

The objectives and tests can be shown, or even issued, to the parent 
who does not want his child **br ainwashed** so he can try them out at home. 

The administrator can also find uses for objectives, for example, 
as measures of his school’s or staff’s performance, that put him more 
centrally in the picture. They would then no longer be some mystic 
thing that today’s teachers have and with which he is out of touch. 

They can become a meaningful tool for him to measure instructional 
performance and to aid in making decisions about programs. If the ad- 
ministrator can be put ’*in the picture” he can find his own uses for 
them. This proved to be an unanticipated form of optimization that 
the project staff uncovered. 

The collections, at the time of the visit, consisted of two main 
groups. The previously published collectionc are mostly cognitive 
objectives in the various disciplines — the reading, 'riting, 'rithmetic 
kind of thing — with some affective collections and some psychomotor 
ones, the latter establishing physical performance to be met. The 
newer ones currently being worked upon, recently finished, or just 
being started, take new directions. 
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One of the published collections handles self-concept. It attempts 
to measure wh£>t the student thinks of himself and of his associates — parents, 
peers and so forth. Conforming with the notion ox the more formal instruc- 
tional objectlvesi this can be used to measure the prevalent attitude among 
students of a class or of a whole school, or possibly as a spot check acrocs 
one or another kind of grouping. 

A more recent collection measures tolerance as an attitude. When 
the endeavor enters this kind of domain, the question arises as to whether 
the items measured always represent true objectives. Is absolute tolerance 
of everything and everybody a viable objective? Because the measures come 
in packages similar to those used to measure performance it seems inevitable 
that people will interpret use of such measures both positively and nega- 
tively, depending on personal viewpoint. 

It could be argued that the tolerance collection lacks a morality 
base. This possible eonnoCation led to the idea of a judgment collection 
to measure th'i cognitive recognition of proper constraints on behavior. 

While this collection is designated "cognitive" it is really close to be- 
coming a bridge between the affective and cognitive domains. For example, 
if a child (or for that matter, any person) observes a conflict between 
persons of obvious differences what does that individual really observe: 
is he concerned with the issues that form the basis of the conflict, or 
is his observation influenced by prejudice based on personal conditioning? 

The judgment collection was developed to assist in restoring or maintain- 
ing a balance that unguarded use of the tolerance collection might en- 
danger. 

A problem with any objective and measure designed to determine 
attitudes is in its construction. Often the purpose is to differen- 
tiate "correct" and "incorrect" responses. The problem is compounded 
by the fact that most people (including children) are preconditioned 
to the notion that any question, any test, must have one correct re- 
sponse and one or more Incorrect responses. 



Quality Control 

This control extends beyond the formulation of the objective state- 
ments and corresponding measurement questions. The notion just discussed, 
that each question must have an "expected" answer, tends to create a dif- 
ficulty in getting honest, i.e., non-conditioned, responses: what the 

individual's personal reaction, thought, belief, or deduction is. 

One step that has been found quite useful is to avoid the use of 
any word that connotes "test" and substitute the concept of "survey," 
supporting this with an attitude that seeks the aid of the participants: 
that they are helping "us" by letting us know what they think, how they 
react, etc. 
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There may be no way to guarantee that these irrelevant factors 
do not interfere with results. But there are ways to determine if 
they are doing so, and to arrive at causes. If a questionnaire is 
too long, for example, the temptation to play the answer -pattern game 
becomes stronger. It is further reinforced if the student has diffi- 
culty understanding the intent of the questions, when his reaction is, 
"What is he asking me?" 

Further positive reliability checks can be run on the objectives 
and measures. Measures that fail to !^ive consistent results need re- 
examining. Beyond this, efforts at quality control can use feedback 
from students and faculty relative to their reactions to the measures. 
A wide variety of mechanisms for checking quality is used in this pro- 
ject as an aid to quality control. 



New Classification Schema 



The third commitment of this project is to devise new classification 
schema. Many questions demonstrate the need for a new way of classifying 
objectives. Finding a simple way to do all these complicated things pre- 
sents a problem. 

Problems . As the objectives are now presented they come in simple 
sets, usually organized by traditional subject-matter discipline and by 
grade level, or some indication of level analogous to grade. As sets 
they are distinguished between cognitive, psychomotor, and affective. 
While a new classification scheme may not be discipline free, it is felt 
that it should break away from the fact-oriented set that use of a dis- 
cipline traditionally develops. It should also provide the user with 
alternative paths rather than coercive, single-channel measures. 

In thinking about the development of a new classification scheme 
a first approach that this project took was in the direction of a simpli- 
fied, computerized access system that would take all the relevant vari- 
ables and enable the user to select examples of objectives and their 
attendant measures. The user was to be provided with directions to 
enable him to construct his own additional measures merely by punching 
up the desired combinations in a computerized system. * This notion was 
dism .d as not feasible at this point. 

Among cognitive objectives, a categorization scheme could be 
built around response mode and cognitive level. Another basis for 
division would be according to user’s needs. Various populations use 
objectives and although no specific analyses had been nin at the time 
of the visit, it was suspected that most users are other than teachers. 
Various level administrators might use them more than teachers, for 
example. 
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All of the available literature on various forms of classification 
for objectives, Instruction, and learning, were carefully studied. The 
Initial step toward a new classification scheme envisioned at the time 
of the visit was the development of a secondary classification or coding 
scheme, possibly as a slip-sheet to be Inserted In existing sets. 

Whatever scheme is finally adopted, it must be applicable across 
the various fields within education. It must also be a scheme that 
school people will appreciate, because its success depends on their 
acceptance. 

A further problem with objectives arises from the traditional 
attitudes with which they are approached, i.e., as absolutes or impera- 
tives. The intent of the objectives already assembled within the Ex- 
change is to provide alternatives, rather than imperatives. A future 
objective that is seen as vital to any new classification scheme would 
be to expand available alternatives rather than to restrict them to im- 
peratives. 

Another condition that hinders optimum use of objectives is the 
homogeneity with which they are traditionally used. There is no sensi- 
tivity about the domain to which the behavior, resulting from instruc- 
tional presentation or whatever, should transfer. This becomes little 
better than the traditional conditioning to "answer questions" with the 
precise response teacher wants: the kind of response where the student 

memorizes a statement which may or may not convey any meaning to him. A 
student may respond "correctly" when presented with the same sentence in 
which a blank is left for one of the words: he can replace the missing 

word. This indicates only that he remembers the words, not that he has 
any appreciation for their meaning. 

This is where the notion of item forms (properly used) can be help- 
ful. The objective specifies the parameters of real concern, gives a 
sample format for question or test item, possibly with six sample 
items, and thus enables school people to generate their own test items. 

Path toward solution . To provide a rationale for the classifica- 
tion scheme eventually identified, the first step will probably be a 
position paper explaining the tentative pragmatic schema. 

From this starting point, procedural steps for sorting existing 
objectives, as well as for ensuring that their format adequately identi- 
fies their individual positions in the schema, will be developed. This 
will provide the mechanism for a "first round" reclassification. 
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Next will come procedural steps for revising existing objectives 
to fit the schema more precisely, and for the generation of new objectives 
and measures to fill vacant categories made manifest by the classification 
scheme . 

Referring finally to the set of output maps, it will be seen that 
they overlap, or nest, although they are drawn independently in Figures 
3 through 7. The new classification schema will fit onto the quality 
control arrangement, to provide an even better means of securing 
quality in all products generated or revised. 
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Chapter VI; Implications for Training 



Common factors in the background of the personnel working on this 
project were experience in teaching and previous background in writing 
(behavioral) objectives. This background was coupled with a personal 
belief in the value of objectives in each instance. 

The project is doing several things that ”cut new ground" — that have 
not been done before: particularly in working toward new classification 

schema and quality control. 

The project has included papers to prepare people working within it 
for the work they must do. This is a quick form of introduction, suited 
to people who have already shown sufficient interest by "coming aboard," 

As the generation of objectives with the potential already demonstrated by 
this project expands in scope other agencies may embark on similar projects. 
This project will have gained sufficient experience to put together a train- 
ing package suited for use in preparing personnel for this kind of work* 

In a sense, this kind of training is already being embodied into the 
generation of products issued by this product. Their concern is to provide 
objectives with specifications for items that will test for those objec- 
tives, with examples, to enable teachers to generate their own test items. 

The original intent of the group was to institute an instructional objec- 
tives exchange to make available objectives and measures generated externally. 

The main reason that the current phase has changed emphasis to internal 
generation was the scarcity of such externally generated objectives, or 
the difficulty in correlating the somewhat randomly distributed samples 
that they did collect. In short, nobody in the field possesses training 
to produce what this project is learning to do "the hard way." 

This led to the development of the procedures now being used within 
this project to improve quality and correlation. 

As the goal of the project is to enable teachers to generate their 
own test items against the specifications spelled out in the objective 
statements, a further step will be to systematize, through the newly 
developing classification schema, the objective statements themselves. 

This will enable "outsiders" to take the next step. This is, in a sense, 
a form of training program that will eventually have wide impact. 

In addition to the more obvious backgrounds for generating the 
objectives (a knowledge and some experience in the generation of 
specific Items and their measures), another need on this project is for 
relatively simple statistical methods. More important than the knowledge 
of statistical methods, however, is the ability to apply these methods 
to the determination of validity and reliability of results obtained with 
the measures as indicative of the objectives they are intended to measure. 
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Appendix: Listing of Output Standards, Tasks, and Enablers 

The following is a list of standards, tasks, and enablers for 
outputs around which interviews were conducted. These statements were 
extracted from discussions with interviewees and were coded into their 
respective category sets. The selected code precedes the statement 
and indicates the following for: 

STANDARDS 



Code J: 


Structure of Standards. 


J-1 


Standards against which outputs are judged, (output 
oriented) 


J-2 


Standards against which processes and/or operations are 
judged, (process oriented) 


Code LM: 


Primary Categories of Standards. 



TASKS 



Code NO: 


Clusters of Tasks. 



ENABLERS 



Code S: 


Structure of Enablers. 


S-1 


Knowledge . 


S-2 


Skill or ability to perform. 


S-3 


Sensitivity or awareness. 


Code UV: 


Primary Categories of Enablers (knowledges > skills, or 
sensitivities) . 



The codes associated with these three categories (standards, tasks, 
enablers) are the same both here in the listing and as pre^^lously cited 
in Chapter III tables. 

Outputs P-45 and greater have parenthetical, compound numerals fol- 
lowing the major identification numeral (e.g., P-06/P-15) . This is 
explained in Chapter III. 



O 
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P-03; A New Classification Schema for Behavioral Objectives 



STANDARDS : 

1 06 School-type people intuitively appreciate the new schema. 

1 01 The cystem Includes objectives applicable across subject fields. 

1 06 Users appreciate the new materials. 

1 08 Obtaining of agreement on the trial schema. 

1 05 Generalizability of objectives is adequate. 

TASKS ; 



m 

01 Read everything that is around pertinent to classification. 

05 On basis of user-need classification , sample user population 

(mostly not classroom teachers) and primary use. 

03 Consider steps to classification: first the issue of a slip 

sheet with existing collections; later reclassified collections, 
with new objectives generated to fit schema. 

05 Try randomly selected samples and classify according to schema. 

03 Analyze objectives by natural language — by structure — to 
see **what falls out”. 

26 Supervise one or two summer people to verify Gagne *s approach. 

01 Draw distinctions between approaches of Gagne and Bloom. 

02 Write position paper. 

04 Write summary statement of alternative measurement procedures. 

03 Specify parameters concerned with item forms (implications of 
various items forms) . 

03 Consider possibility of using series of item forms with complete 

specifications instead of objective statement. 



ENABLERS: 



S UV 

3 18 Sensitivity to opposition attitude about ^objectives .” 

1 05 Knowledge of Gagne *s conceptual work. 

3 30 Sensitivity to questions school people would ask. 

2 09 Applying subjective judgment in achieving criteria. 

2 11 Ability to disregard own bias until empirically tested. 
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P-20: Technical Paper //3 



standards: 



J 

1 07 When I felt it was clear to me I put it in print. 

1 30 When I received no negative response of questions from the first 

two readers I assumed it clear. 

tasks: 



m 

04 Write Technical Paper for new people to help them generate 

new collection items. 

02 Mull over what is necessary to generate new collection items. 



enablers: 



i yv 

1 03 Knowledge of definition requirements of behavioral objectives. 

2 35 Skill in presenting my ideas in clear form. 

3 02 Sensitivity to ignorance or naivete of new employee to 

certain techniques and methodologies. 



P-21: Quality Control Schema for Tryout Version of Collections 



STANDARDS : 



J 

1 13 Content judged valid by group consensus of the 3-man directorate. 

1 24 Level of reading, conceptual sense, are all appropriate to target 

audience as judged by group consensus. 

1 26 Measurement items and objectives match. 

1 26 Match between objectives ^d philosophical statement introducing 

them. 

1 24 Appropriate language, grammar, has been used. 

TASKS : 



m 

24 



For the cognitive objectives which had excellent structure we 
judge the definitions in terms of content \ .lidity. 




J 
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03 We define the affective objectives in terms of the instruments 

we developed to measure them. 

01 Reviewed the literature to try to understand what was meant by 

affective concepts . 

24 We determined if the definitions of objectives contained words 

appropriate to the grade level of the objective (K-12) . 

22 Have tryout version dittoed. 

24 Have measures tried out on kids for readability. 

24 Refer to experts for content appropriateness, accuracy. 

24 Refer to teachers for suitability for age level. 

06 Discover sentences susceptible to misinterpretation. 

06 Give document a final hard review. 

06 Review introductory qualifying statements. 

06 Eliminate any offensive items. 

06 Check scoring for accuracy of indications. 

06 May make changes even after empirically tested, of wording '*bad.** 

29 Look at all the questions someone might ask, get into a position 

to be able to defend all that is in the collections. 

ENABLERS : 

S 

1 23 Knowledge of vocabulary level of students and teachers at 

various . grade levels . 

3 02 Sens! ti vi ty to teacher ’ s problems such as time pressures etc. 



P-26: Quality Control Schema for Preferential. Data on Objectives 



STANDARDS : 

J ^ 

1 13 Consensual judgment that all items are acceptable . 

1 05 Utility by teacher. 

1 05 The teacher or user can use data and apply as they desire. 

1 04 Clarity, based on personal judgment. 

TASKS; 

NO 

05 We put the objective through preliminary tests . 

06 Cull or filter objectives for gross acceptability. 

24 Test the objectives by having the different sets of objectives 

rated ty different groups. 
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06 Confirm validity by comparing the initial objectives with 

rewarding of objective. 

06 Compare rewarded objective with test item, etc. 

24 Assure reliability be having everyone working on the task 

at one time or another. 

ENABLERS : 

S 

1 18 Knowledge of personal judgmental limitations . 

3 01 Sensitivity to other points of view. 

1 06 Knowledge of teachers' problems. 



P-41: Review of Proposed Classification Suggestions 



STANDARDS • 



J IM 
2 36 
1 05 
1 04 

1 11 
1 12 

1 12 



TASKS : 

21 

06 

21 

06 

06 

02 



People selected were capable of generating alternative hypotheses. 
Utility to teacherc of the classification schema. 

Classification schema for objectives simple and easily understood 
by teachers . 

Classification schema for objectives applicable to ES '70 school. 
Classification schema for objectives contains some classification 
of behaviors. 

Classification schema for objectives contains some classification 
of content. 



Select consultants capable of reviewing suggestions for clas- 
sification schema. 

Consultants write down further suggestions for classification 
schemes for objectives . 

Secure services of good consultants. 

Review for the consultants the classification of objectives schemes 
which had not worked. 

Suggest alternatives for classifying objectives . 

Write position papers to define the dimensions of classification. 
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ENABLERS: 

S 

1 12 Knowledge of expert problem solvers in the field. 

3 01 Sensitivity that reputation could attract talent. 

3 03 Sensitivity to needs of consultants they want to learn. 

3 39 Sensitivity to limits of one day consultancy . 

3 14 Be willing to settle for the best results possible rather 

than the best possible results. 

3 40 I feel obligated to follow through on the original request 

for classification for ES *70 schools. 

3 16 That institutional affiliations are not important in judging 

a person. 

3 34 I would consider an unknown person if a known reputable persjn 

told me about him. 

3 25 Aware of other people*s perceptions. 



P-45: (P-06/P-15) . Rationale Statement for American Government Objectives 



STANDARDS : 

No information collected under this heading. 
TASKS : 

m 
04 
01 
02 



ENABLERS : 

S W 

1 01 Understanding of subject, e.g., separation of powers, what 

sovereignty' j.s . 



Outliie terminal-type objectives of conceptual nature. 
Refer to library, personal authorities on subject. 

Wrestled with concepts difficult to convey to students, or 
to test for attainment of objective. 
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P-46; (P-08/P-13). Tryout Version of Cognitive Judj^ent Measures 



STANDARDS : 

No information collected found under r.his heading. 

TASKS : 

m 

22 Have accepted version of measure reproduced in mimeograph 

or other temporary form. 

'29 Secure district approval for use of tryout objectives in schools. 

ENABLERS : 

s m 

1 12 Knowing the people necessary to secure approval and exposure. 



P-4 7: (P-08/P-14). ^nhouse Version of Cognitive Judgment and Measures 




STANDARDS : 

J IM 

2 17 General reaction from students and other participants 

tnat ”this is a fun test” indicating relaxed attention. 

1 26 Relationships between objectives and related test questions 

validated by a jury of experts. 

1 20 Reliability studies conducted between responses reflect adequacy. 

2 17 Students very cooperative — like ”doing” the tests. 

TASKS ; 

04 Key test items to the measure for which they are applicable. 

04 Produce a complete set of tests based on criteria development 
included objectives P-48. 

05 Run tests with cooperation of local hJgh school continuation 
school in the community and with student teachers in UCLA. 

06 Make revisions based on feedback from tests (what they tell 
us) . 
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05 Conduct tests with samplings of students in cafeteria of 

schools • 

ENABI^RS : 

S W 

2 23 Ability to convince students and other participants that the 

tryout is '^helping us." 



P-48: (P-08/P-15). Rationale for Cognitive Judgment Measures 



STANDARDS : 

J IM 

1 19 Rules of logic after Robert Ennis are in evidence. 

1 16 Watson-Claser critical thinking best rationale in evidence. 

1 06 Whether others on staff and in field "buy" the rationale. 

2 l5 Interviewee endorsement for the idea made further work 

possible, encouraged progress. 

TASKS : 

NO 

02 React with staff about the need for these objectives. 

02 Develop logic as basis for "judgment making." 

02 Develop number of specific objectives to have comprehensive tests. 

33 Decide to include recognition of logically valid statements. 

33 Decide to include tests to separate logical ability with and 

without emotional involvement (using emotionally loaded questions). 
04 Include reference to the cognitive judgment collection in cover 

letter issued with attitude measure collection (Tolerance) . 

04 Develop rationale for objectives to be included in collection. 

03 Suggest "propaganda analysis" as component (s) of cor.lections 

to establish, e.g., difference between validity and truth. 

ENABLERS : 

5 W 

3 31 The need for component as offset to effects of another 

component . 
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3 16 Realization of a problem where no value reference exists. 

1 06 The need illustrated in the illogical reactions of people. 



P-49: (P-08/P-18) . Staff-generated Objective Statements for Cognitive 

Judgment Measures 



STANDARDS : 

1 LM 

1 19 References used are relevant to intent of these objectives . 

1 13 Objective statements approved by staff. 

TASKS : 

NO 

04 Using standards of logic and morality, generate key items 

to measure. 

06 Test list of key items for completeness. 

01 Look up various sources (e.g., in library) on "judgment” 
(subject) . 

02 Address question as of form, how to measure "intolerance." 

01 Look for definition of "inference" (none satisfactory found). 

02 Seek definition from members of Philosophy Department (they 
could not agree). 

02 Consider creating an objective on fallacies (still undecided at 

time of interview). 

04 Choose proposition sets appropriate to student level. 

06 Eliminate for time being propositions in perception that 

require pictorial presentation. 

06 Discard irrelevant aspects of judgment, e.g., arc. 

01 Analyze information to discover if it is relevant to a given 

problem. 

ENABLERS : 

1 UV 

2 18 Recognizing logical validity cf statements. 

1 03 Background in behavioral objectives. 

3 42 Enthused with potential of behavioral objectives. 

3 43 Finds objective writing a challenge — much harder to formulate 

than in mathematics. 




P-50: (P-08/P-19). Staff-generated Test Items for Cognitive Judgment 

Measures 

1 



STANDARDS- 

1 LM 

1 22 Students were able to respond appropriately to inferential test. 

1 21 Items selected are those on which the quality of student 

responses was rated as high. 

1 13 Typed up item sets approved by staff. 

TASKS: 

NO 

04 Generate some test itmes based on standard class logic 

propositions and on assumption recognition. 

04 Work up questions following examples, taking the principles 
from Robert Ennis. 

02 Wrestle with problem of creating test items that are simply 
stated, unambiguous, yet which require the inference to be 
made to answer correctly. 

05 Ask teachers to report words with which students experience 
difficulty . 

ENABLERS: 

_s m 

2 35 Skill at wording questions so as to determine if objective has been 

attained, without "cluing** student. 



P-51: (P-12/P-29) . of Data on Usage of Objectives and 

Measures for Usage Pamphlet 



STANDARDS: 

1 LM 

1 22 Effectiveness in achieving acceptance by various classes 

addressed in this research. 
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TASKS ; 

01 Use Popham’s paper (1/70) ‘'Potential Uses of Objectives" as 

basic starting point. 

05 Analyze data on use of attitude measures. 

05 Check if attitudes drop off in class of "good" teacher. 

05 Determine period of greatest cognitive gain in students. 

05 Check effective testing results against state-wide standardized 

tests . 

29 Use observational fundings as inputs to persuade teacher and 

administration to use objectives. 

07 Show teachers how to become students of their own teaching by 

using objectives, 

07 Show principals how to use tests as managerial aid: puts 

principal "in it." 

07 Practice communicating the function of objectives and tests 

to parents, to offset "brain washing" propaganda. 

24 Plan tests to determine effects of letting students know objectives. 

ENABLERS : 

S W 

3 03 Sensitive to needs of various groups: administrators, teachers, 

students, parents . 

3 04 Sensitive to involvement of various groups. 



P-52: (P-21/P-24). Quality Control Schema for Tryout 

Version of Cognitive Objectives 



STANDARDS : 

J IM 

1 07 Intuitive feel for whether or not the job was well done. 

1 13 Reaction of the writer and other reviewers to the collection 

was good. 

1 06 Teacher comments from field review for the sales version of 

the collection were good. 
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TASKS : 

NO 

31 

31 

24 

06 

02 

06 

02 

24 

31 

06 

31 

06 

24 

06 

24 

26 

22 

06 

06 

04 



Receive a collection of objectives with one test item for each 
from the colleague who had generated them. 

Receive a memo of what to look for in the objectives, test 
items and their interrelationships in the way of review. 

Read the objectives/test items with memo notes in mind to check 
quality. 

Write comments on separate sheets for the writer to review later. 
Confer with colleagues regarding questions which arise in the 
review f)rocess. 

Rewrite items for which felt (a) revision was necessary and (b) 
self was capable regardless of whether actually am. 

Interact with colleagues on items which seem to be worth 
discussion as opposed to a written comment. 

Return the collection and comments to colleagues for review 
and consideration . 

Meet to discuss questionable items to insure better understanding 
and agreement on them. 

Revise the collection based on comments and discussion for review 
by a subject area specialist. 

Carry collection to a subject area spcialist for general review 
of subject coverage and accuracy. 

Review and write comments on collection for subject coverage 
and accuracy. 

Review comments of subject area specialist for changes necessitated. 
Update the collection according to comments of subject area person 
and oyn judgment. 

Send a copy to the Project Director or Co-director for 
final review. 

Review the completed collection to give approval for printing 
or a tryout version of the collection. 

Send the final copy to the typist and print shop facilities 
for reproduction. 

Review the collection for suitability with project purpose and 
write comments. 

Update the collection as required by Director comments for this 
tryouc version of the collection. 

Add five tc six additional test items for each objective. 



ENABLERS : 



£ M 

2 19 Ability to (read an objective and) visualize the domain to 

which it is applicable so that other test items may be generated 
within that domain regarding that objective. 

2 17 Ability to understand what the person is saying in a stated 

objective . 

3 10 Sensitive to what objectives will be accepted by the Project 

Director as not being too specific or too general. 
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2 lA Decent English in order to write ob joctives/test items. 

1 05 Knowledge of what a student at a given grade level can and 

cannot do. 

1 01 Familiarity with a school curriculum so that one can see how 

the objectives could be adopted into that curriculum. 

3 17 Sensitivity to the person who wrote rhe collection and their 

work efforts in generating it. 

3 02 Sensitivity to the grade level mentality, e.g., 4th graders 

need things lined up physically so they know where to answer. 
3 02 Sensitivity to 4th grade vocabulary. 

2 01 Teaching experience in 4th grade. 

3 18 Lack of subject area background so can be more objective. 

3 01 Considered to be a critical person, sensitive i;o that about 

myself. 

3 07 Sensitive to ** terminal** type objectives. 



P-53: (P-21/P-25). Quality Control Schema for Tryout 

Version of Affective Measures 



STANDARDS : 

1 09 Feel like have weeded out ambiguities. 

1 12 Results from testing for sales version collections show 

desired ends met. 

1 13 Reactions of other reviewers to the collection is positive. 

TASKS ; 

NO 

31 Receive measure from the colleague who generated it for 

review and comments . 

06 Review the items in the measure for ambiguities and grammar, 

revising on the cards (one item per card). 

05 Take the test pretending to be a student who lik^s school, to 
see if answers would be positive. 

24 Interact with colleagues on results of trial test for suggestions. 

01 Review literature provided by staff or found in library to 

sensitize self to good attitude statements and levels of 
offensiveness . 

02 Go through the items with colleague(s) to look for duplication ^ 
etc., and brainstorming for comments. 

24 Make piles of good and bad items according to one’s own judgment. 

06 Reword bad items where doing so might improve the item. 
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24 Select the best items until a manageable number (100) of items 

are s, elected for the measure. 

22 Present the cards containing the items and format instructions 

to the typist for a draft. 

24 Give a copy to the Program Director for review. 

06 Make any necessary changes in the measure necessitated by the 

Program Director’s comments for the tryout version of the 
measure . 

22 Send the final version to the typist for a final copy of 

the nieasure. 

04 Put together the various ineasures of the collections for 

publication. 

04 Write introductory materials for the collection. 

22 Send the final version to the reproduction facility for 

publication. 

ENABLERS : 

s uv 

1 03 Knowledge of what make'^^ a good attitudinal measure item from 

reading in the field: Handbook in Social Psychology » Thurstone, 

etc. 

1 04 Familiarity with attitudinal measures from a social psychology 

class . 

3 02 Sensitivity to age level and capabilities of students. 

3 16 Sensitivity to the "value problem" among students from varying 

backgrounds . 

1 08 Know the overall framework of the measure and its boundaries. 

3 09 Sensitivity to different possible ways to interpret test items. 

2 17 Ability to understand what the person is saying in a test item. 

3 27 Sensitivity to what is acceptable to the Project Director. 

3 17 Sensitivity to the person who wrote the measure items and their 

work efforts in generating it. 

3 02 Sensitivity to grade level mentality, e.g. , 4th graders need 

things lined up physically so they know where to answer. 

3 02 Sensitivity to grade level vocabulary. 

1 03 Know how to write test items. 



P“54: (P-04/P-14). Inhouse Version of Sociology Objective 

Collection 



STANDARDS : 

J LM 

1 01 Feel field is covered in each area as additional review of 

n.C 
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1 13 
1 16 
1 07 

1 07 
1 06 

TASKS: 

m 

01 

01 

01 

04 

01 

02 

02 

24 

06 

31 

06 

01 

31 

01 

01 

04 

01 



31 

01 

01 

06 

04 

06 



literature fails to turn up new areas. 

Approval of colleagues, Project Director , subject area specialist . 
Coverage compares favorably with college leve] materials. 
Objectives don*t seem trivial (to me). 

Subjective judgment based on own knowledge of sociology. 

Approval of the objectives by teachers in the field. 



Go to curriculum lab at UCLA and check out everything with 
the name sociology on it. 

Go through the elementary social studies mateiials to determine 
degree of coverage of sociology. 

Abstract that which seems useful. 

Write objectives based on the abstractions from literature. 

Look over another collection as a prototype. 

Consult with colleagues on whether an objective could be 
considered terminal or whether it is trivial. 

Consult with colleagues on appropriateness of an objective for 
particular grade levels. 

Take collection of objectives to subject area specialist for 
comments on subject accuracy and coverage. 

Read subject area specialist comments and recommendations re. 
subject coverage and accuracy. 

Discuss comments with subject area specialist for better under- 
standing. 

Tally the number of objectives within each sociology area to get 
an idea of extent of coverage of the areas. 

Decide how many areas to cover within sociology for grade level 
and depth of coverage for each area and level. 

Add areas suggested by subject area specialist. 

Write down all important topics covered in literature. 

Reduce topics to be addressed to about five for each sociology 
area. 

Select objective to cover each topic in each area. 

Compare elementary coverage with college coverage and include 
areas of coverage which feel are relevant but not already 
covered in objectives. 

Discuss need for additional materials with colleagues and get 
a list of sources. 

Write to prospective sources from discussion and those picked 
up in search of the literature. 

Read literature which comes in that is relevant and may contain 
helpful information. 

Reduce objectives to about 30 or 35 terminal objectives boiling 
some down where necessary to maintain coverage. 

Add a sample test item which tests successful completion of the 
objective where this has not already been done. 

Review the collection for general overview before sending to a 
collegian for first formal review. 
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24 Send the collection to a collegian to review for a variety of 

known points for criticism. 



ENABLERS : 

S W 

3 13 Sensitivity to terminiology i.e., what is absolutely essential 

without turning students off with vocabulary. 

1 17 Knowledge of what to do with terminology once essential ones 

have been selected. 

1 01 Knowledge of what books are available in elementary level 

social studies through search of card catalog and discussion 
with subject area specialist. 

1 01 Knowledge of elementary sociology (introduction to social course) . 

1 23 Knowledge of level of materials for grade levels of kids. 

3 02 Sensitivity to what grade levels can do what. 

2 20 Skill in objectivity due to lack of specialization in sociology. 

2 18 Common sense to be able to tell whether an item or information 

would be useful for a kid to know. 

2 24 Skill in writing significant objectives, i.e., that cover large 

bodies of information and at the same time be behavioral and 
include specifically what you mean. 



I 

P-55: (P-27/P-29). Data Summary for Affective 

Measures Quality Control 



STANDARDS : 



J 

1 05 
1 05 

1 09 

2 14 
1 13 



1 07 



The summaries are useful in writing the data report. 

The summaries are useful in punching data for correlation analysis. 
The tally checks show high degree of accuracy. 

All p::e*«/ious steps in the quality control process went well. 
Acceptance of the data report containing the summaries information 
by the person in charge of the revised sales version quality 
control for affective measures . 

Graduate students in education do good job of hand tallying. 



TASKS : 



m 

21 Meet with volunteers to do tallying to give instructions to 

them. 

Train new people according to a tally system developed by colleague. 
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21 Demonstrate how to punch answer templates from pages in 

collection booklet. 

05 Tally test items until tired. 

05 Hand tally intermediate level tests with colleague. 

29 Interact with colleague regarding Constance of answers and 

comments written on tests (as instructed). 

24 Check every five tests scored by a tallier for accuracy such 

that 9 of 10 were not off more than one point. 

21 Draw a grid to show talliers how the analysis designed to 

check correlation . 

21 Explained to talliers what the data v;ere about and what item 

discrimination was. 

21 Show talliers the actual test they will tally. 

06 Perform a variability check on samples of tests. 

05 Tally class totals by hand for item discrimination. 

05 Write item discrimination totals into a sales version 

collection beside the particular item for the different 
classes . 



ENABLERS : 

S W 

1 03 How to hand tally. 

2 02 Skill in working with people. 

3 21 Sensitivity to accuracy of hand tallies. 

3 21 Sensitivity to meaning of item discrimination and correlation 

analysis/data. 

2 19 Skill in organizing. 

1 03 Knowledge of the field of attitudinal measures. 

1 24 Knowledge of which items are to be tests to score. 

1 22 Knowledge how to operate template punch. 



P-56: (P-27/P-33). Data Analysis Procedures for Affective 

Measures Quality Control 



STANDARDS : 

J 

1 05 The correlation summary useful in writing the data report. 

2 14 All previous steps in quality control process went well. 

1 13 Acceptance of the data report containing the correlations by 

the person in charge of the revised sales version quality control 

for effective measures. 






153 



598 



TASKS : 

01 Read the correlation analysis chapter in a book by Project 

Director . 

05 Compute correlations on desk calculator. 

31 Ask lor and receive instructions for running the desk calculator. 

05 Compute correlations on machina . Punch in five scores for 

machine analyses. 

05 Write a list of tests correlated and the correlations from the 

machine. 

04 Type ditto master for correlation summary for reproduction and 

distribution. 

ENABLERS : 

1 uv 

1 22 Knowledge how to type a ditto master. 

1 22 Knowledge how to run a ditto machine. 

1 08 Knowledge which tests were to be correlated from the grid 

scheme . 

1 03 Knowledge of concept of and procedure for correlation analysis. 

1 22 Knowledge how to run statistics machine. 



P-57: (P-26/P-29). Data Summary for Cognitive Objectives 

Quality Control 



STANDARDS: 

J LM 

1 06 Acceptance by users. 

1 05 Utility by teachers. 

1 13 Consensus of staff. 

1 18 Visually nice. 

1 04 Easy to understand. 

1 04 Concise. 

1 05 Useful. 

1 08 Consensus of sampling of people. 

1 28 Useful — not too cumbersome — judged by consensus. 

TASKS: 



We correlated between group ratings. 
Decide on format of summary in book. 





m 

05 

33 



04 



Write explanations of groupings and data for each as well 
as explanation of how it will be presented. 

05 Star the objectives, i.e., four stars for excelxent, one star for 

poor, based on rating. 

05 Retest an objective each time it is changed. 

24 Ask raters **is this clear to. ..." 

31 Show rating results to teachers in classrooms. 

24 I restrict myself to a 2-page format. 

03 Try different formats for data summaries. 

ENABLERS ; 

s m 

1 02 Knowledges of statistics (simple statistics). 

2 19 Ability to organize. 

2 20 Ability to think logically. 

2 20 Being practical — not esoteric. 

2 05 Being in systems analysis techniques. 

1 03 Knowledges of testing procedures. 

3 04 Interaction with people. 

1 06 Knowledge of educational procedures. 

1 06 Knowledge of classroom procedures. 

1 06 Knowledge of how teachers do in school outside of classroom. 

2 14 Writing ability — clarity. 

2 26 A skill in searching out new formats. 

2 38 Skill in applying knowledge to real world problems. 

2 10 Applying statistics to real world situations. 



P-58: (P-26/P-30). Data Sources for Cognitive Objectives 



STANDARDS : 

J ^ 

2 36 The people were selected because I knew them and they were 

easy to contact . 

1 05 Usefulness/simplicity of categories. 

TASKS: 

m 

30 As a result of reading Future Shock we included futurists 

as data sources. 
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23 1 select n council of the future to include four graduate 

students, three professors, and a housewife?. 

04 We included questions in questionnaire about community. 

ENABLERS: 



s uv 

1 02 Knowledge of educational curriculum theory. 

3 18 Sensitivity to opinion that behavorial objective rejected 

by "humanists." 



P-59; (P-26/F-31). Data -gathering Procedures for Cognitive 

Objectives Quality Control 



STANDARDS : . 

J 

1 03 Time spent in class — if too long, teachers reject. 

2 24 We will decide on which method by cost/benefit guess. 

TASKS : 

06 Change the objectives number system to a letter system. 

06 Change the name from test to survey. 

05 Where student responses went 000 (111 or 222) ask them why. 

05 Ask students for an essay response about community to obtain 

vocabulary (word community thinks in) used by that group as well 
as to determine future categories for that group to be divided 
by etnnic, economics etc. 

05 Ask teachers for an essay response on community and students to 

determine vocabulary for later IBM type questionnaire. 

24 Solicit rating response of futurists. 

05 Futurists rate objectives after reading "Future Shock" and 

pretending they are living in 1980. 

03 Generate several alternative ways of operating mail system. 

03 Develop methods for collecting data by mail. 

05 Try out the methods to see which gave best results (i.e., 

most returns). 

05 Students write down words they do not understand. 

29 Obtain permission of the principals to enter the school. 

05 Obtain data from other geographical areas through a mail 
system of collection. 

06 Observe the children's action on this mail — data collection 
procedure to determine reasons for poor return. 

Confirm with staff adequacy of methods. 



24 
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ENABLKRS : 

S W 

3 22 Sensitive to reaction of students wh(:n principal present in room 

during data collection. 

3 22 We never used the word "test" because it creates anxiety in 

students . 

3 35 We were careful about emphasizing ratings rather than rankings. 

3 22 We had to watch out for students going beyond instructions. 

2 31 Ability to place myself in position of teacher. 

3 02 Sensitivity to fact teachers get large quantities of mail, 

surveys, etc. 



P-60; (P-26/P-32). Data Compilation Instrument for Cognitive 

Objectives Quality Control 



STANDARDS; : 



J 

1 30 
1 28 
1 22 
1 30 

1 31 

2 24 
1 32 
1 28 
1 12 
1 12 

TASKS : 

NO 

04 

29 
04 

04 

30 
30 
01 

06 

06 

33 

05 

06 



Few remarks filled in (no response) . 

Understandable — clear as judged by past use of raters. 
Inappropriate response patterns are not evident. 
Teacher judgment that instruiuents are not too long. 
Questions are legal (e.g., race, etc.). 
Cost/Effectiveness of test instrument acceptable. 

Good reputation of test maker. 

Instrument is practical for use with kids. 

Instrument is mailable. 

Instrument is computer convertible. 



We write a set of directions to give importance to instrument. 

We explain a five point scale in instructions. 

We include a six point scale (include zero) in answer sheet. 

We rewrite objectives at lower level where appropriate. 

I went to IBM and asked to look over all of their test instruments. 
I asked IBM who their competitor was and went there. 

Determine if test instrument is computer convertible (Optical 
Scanner, Punch card etc.). 

Revise questionnaire if (don’t know) marked excess. 

Shorten questionnaire if an appropriate response pattern occurs 
at end. 

Decide whether to design or buy test format. 

We try out different formats. 

Shorten questionnaire at request of teacher 
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at enJ. 

Decide whether to design or buy test format. 
We try out different formats. 

Shorten questionnaire at request of teacher. 



Sensitivity to costs of instruments. 

Ability to place myself in position of teacher. 

Ability to search out new test instruments (visit IBM). 
Knowledge of test instrumentation. 

Inquisitivei ess . 

Knowledge of resource availability. 

► — 



P-61: (P-29/P-33). Daua Analysis Procedures for Cognitive Ob- 

jectives Quality Control 



STANDARDS : 

J m 
1 11 



TASKS : 
05 



05 
05 

05 

06 
06 
05 

04 

01 

05 

ENABLERS: 

SOT 

1 02 Knowledges of statistics — methods. 

, 0 '-. 




Aopropriate statistical procedure for data according to 
per s onal j ud gmen t . 



Compare rating of objectives obtained under different 
circumstances, e .g. , objectives , rev;ritten objectives , ob- 
jectives rated with measure as part of rating judgment. 

We took the rating out and did a ranking on them. 

We took the ranking and did a correlation on them. 

We correlate data by paired comparison. 

We did the Pearson Product Moment correlation. 

Review the data for completeness. 

List the characteristics of ray data. 

I found in a statistics book which procedures were appropriate 
for that data. 

Conduct library research to find information and paired 
comparisons . 

Perform analysis on nonzero numbers in data. 



33 

05 

06 

enablers: 



23 

31 

26 

03 

41 

07 
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1 01 Knowledges of how to calculate. 

1 03 Knowledge of defeciences of each statistical method. 
3 21 Sensitive to "appropriateness" of procedure to data. 
1 03 Good knowledge of experimental design. 



P-62: (P-07/P-14) . Inhouse Version of Attitude Obj ectives Collection 



STANDARDS : 

J 

1 13 The criteria for acceptance for content validit> arrived 

at by personal collegial judgment. 

TASKS : 

NO 

01 We decided to work on self-concept beca ise it was dictated 

by Tit^c III funds. 

01 We reviewed literature to see how others dealt with attitude 

measures . 

31 We held a dozen meetings with selected staff to just talk 

about how a youngster would feel about these things — just 
to clarify our own thinking. 



ENABLERS : 

S W 

1 24 Knowledge that we were dealing with group data rather than 

individual data. 

1 08 Knowledge of alternative hypotheses. 

3 24 Sensitivity to how students can "con" the tester or fake 

the test. 



P“63: (P-22/P-24). Quality Control Schema for Sales Version 

of Cognitive Objectives 



Ibo 

o 

ERIC 
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STANDARDS: 

A. M 

2 08 Judgments were accepted from adults — not students. 

1 13 Content judged appropriate by control consultants. 

TASKS; 

24 Subject matter experts and practicing teachers judge the 

collection content validity . 

24 Content consultants accept or reject items. 

24 Teachers leviev; items and write criticisms. 

24 Consider alternative methods for increasing returns on 

questionnaires . 

05 Send out questionniare forms to cooperating schools. 

05 We sent out data summary forms to cooperating schools. 

05 Analyze data returned from cooperating schools. 

ENABLERS ; 

s jyv 

3 02 Sensitivity to teacher's busy schedule. 

1 27 Knowledge of potential cooperating schools. 

2 23 Skill in ootaining cooperation with schools. 



P-66: (P-27/P-23). Rough Draft of Data Report for Revised Sales 

Version Attitudinal Measure Quality Control 



STANDARDS : 

J m 

1 07 Feel competent at writing. 

1 12 Presented the real data. 

1 19 Could explain why correlations were what they were. 

1 05 Could make recommendations based on the results. 

2 14 Confidence in the accuracy and acceptability of all previous 

quality control steps . 

1 13 Acceptance of the data report by the person in charge of 

the revised sales version quality control of affective measures. 

2 14 Felt good because didn't just rely on statistics for results 

but iicluded intuitions from examining the test program and 
score sheets themselves . 

1 01 Testing design included all grade levels and some variety 

in socio-economic backgrounds . 

V, , 166 
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TASKS : 

NO 

02 

04 

05 



06 

04 



04 



06 

04 

30 



06 



ENABLERS : 

s _yv 
1 08 

1 03 

2 14 

3 21 
3 31 
3 22 
3 21 

1 24 

3 22 

1 07 



1 03 
3 06 



Organize Ideas according to individual tests so that each would 
be considered for correlations and item discrimination. 

Present the correlation and item discrimination data in 
writing for each test. 

Determine inferential reasons for high or low correlations 
based on (a) knowledge of the tests themselves and what they 
are supposed to measure and (y statistics which had been computed. 
List item discrimination problems, if any, for each test. 

Write reasons for item problems based on (fO letters from 
teaching, (^) written comments on tests by students, (O 
inferences from having helped tally or having revised groups 
of tests and apparent problems students were having such 
as misunderstanding instructions . 

Write a summary to the paper which includes over all inferences: 

(a) should have had subscale correlations as they are the 
unit of measure and tests consist of different and varying 
Nos. of subscales, No one test can measure ^'school 

sentiment” or ”self concept”; need multiple testing. 

Reread to insure readabili ty , spelling, etc. 

Type and Xerox rough draft of the data. 

Give a copy of the data report to collegian in charge of 
revised scale version quality control for attitudinal 
measures • 

Rewrite some items which were bad and which felt capable of 
rewri ting. 



Knowledge of the attitudinal measures tested and what they 
are supposed fo measure and how they do that. 

Knowledge of the field of attitude measurement — course 
work, reading, on-the-job discussion . 

Skill in writing. 

Sensitive to the data and its meaning. 

Sensitive to the tests and what and how they laeasure attitudes. 
Sensitive to the kids and how they react and why. 

Sensi.tive to the meaning of item discrimination and correlation 
analysis data. 

Knowledge of which tests were used and which were correlated 
with which. 

Sensitive to student remarks to point out ambiguities and 
difficulties they would ignore in normal testing. 

Knowledge of the idea of rationale behind the measures as 
well as the concepts and how to measure it as presented in 
staff meetings . 

Knowledge how to write good test items and criteria for good 
test items. 

Sensitive to the value of the report to the project. 
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The reader should refer to this number when making 
use cf the reader’s GUIDE to the profiles. 
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Chapter I: Overview 



This overview presents a brief synopsis of the REACT Project as an 
introduction. This is elaborated by a discussion of the objectives, ra- 
tionale, and significance of the project and the context in which the 
project operates. 



Synopsis of the Project 

Title: Relevant Educational Application of Computer Technology. 

Responsible Institution: Northwest Regional Educational Laboratory. 

Funding Source: U.S. Office of Education. 

Funding Duration: July 1, 1970 to June 31, 1971. (12 months) 

Observation Date: September 1970. 

Present Stage of Development: Initial -Stage . ^ 

RDD&E Focus of Project: Educational development. 

Expected Outcomes: 



1 . Student instructional units . 

2. An administrative and teacher training unit. 

3. A public relations presentation demonstration package. 

4. A revised administrative and teacher training manual. 

5. An application package for MIDAS. ^ 

6. A plan for the future of the project. 

7. A clearinghouse for educational applications for computer 
technology . 



Level of Funding and Duration: Medium. (level 4 of 7 levels) 



Agency Setting: Regional educational laboratory. 



Staff Summary (Current): 

Total Full Time Equivalency 
(in man years) : 

Number of Personnel Assigned: 



Profession al 

5.25 

6 



Support 

1.0 

1 



Professional Specialities 

of Staff (interviewees only): management; research and devel- 

opment; programmad instruction; 
curriculum development. 



ERIC 



^ Initial-stage under the current funding — developmental ly, REACT is 
in a second phase. 



2 



Computer simulation of school administration (see **Ind 






Outputs") . 
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Objectives, Rationale, and Significance of che ?i-oject: 

The general goals of the Relevant Educational Application of 
Computer Technology (REACT) Project are to develop, field Lest, and 
revise instructional packages which are designed to provide high school 
administrators, t’?achers, and students with experience and under- 
standing in using computers. 

The primary outputs being developed to reach this end are two 
field tested and revised instructional systems (developed during a 
previous project) which include 70 curricular and administrative 
units. Most of che curricular units are in the fields of mathematics 
and social science. These packages are designed to familarize ad- 
ministrators and teachers with the use of computers. Additional out- 
puts include a less expensive package designed to provide computer- 
extended educational experiences for students; and a simulated, com- 
puterized, miniature school system (midAS) and its administrative procedures. 
The latter functions with a limited data base to provide experience 
and aid to persons dealing with resource allocation problems in school 
administration . 

An additional goal is the designing and implementation of a 
clearinghouse for instructional units developed by agencies through- 
out the country and designed for use by students in classrooms to 
provide them with computer-extended education. 

REACT is in its second phase. The final products of the first 
phase were two instructional systems, containing 70 curricular and ad- 
ministrative units, which were aimed at familarizing high school admin- 
istrators and teachers with the possible uses of the computer in dealing 
with administrative and classroom problems respectively. These systems 
were untried and untested. They had been reviewed by experts in com- 
puter usage, mathematics, and social science curriculum areas for 
general validity of the contents . How the units would function when 
tried in actual instruction of administrators and teachers was unknown. 

The present REACT staff, most of whom had been involved in the 
first phase of the project, were eager for and were committed to inten- 
sive testing and revision of these packages. When funds became avail- 
able in July 1970, tentative plans and contacts had already' been made 
by the project staff for selection of testing sites and development of 
a comprehensive evaluation design. The REACT staff are convinced of the 
merit of the computer as a classroom and administrative tool, and the 
strength of their conviction was displayed by the rapidity with which 
the project took on a momentum as soon as it was funded. 

By the time this project was contacted in September 1970, three 
test sites using three different modes of presentation of the instruc- 
tion had been definitely selected and arranged for, and in two cases 
testing had begun. 

Intensive computer-assisted analysis of test questions and responses 
generated from the 70 instructional units had been arranged with 



1T3 



613 



the New York Institute of Technology. Similar evaluation of all the 
presentation packages, both first phase and new packages (written under 
the present REACT contract), will be accomplished at various testing 
sites . 



A significant difference between the thrust of this program and 
programs directed toward computer-assisted instruction (CAI) can be 
seen in the role of the computer itself. In CAI packages the computer 
is the teacher; in REACT, the computer is a problem-solving tool. This 
philosophical difference results in the development of entirely differ- 
ent educational packages. The REACT staff emphasizes that this project 
is designed to provide computer-extended education with the computer 
functioning as a fact calculator, a simulation device, or an element 
in a gaming program. These uses do not have as their emphases the re- 
placement of or partial substitution for human teachers. These have 
traditionally been the emphases of CAI programs. 

REACT instructional packages do not necessitate fundamental changing 
of educational systems; instead they require the addition of increased 
numbers of terminals in school systems which possess computers. As more 
and more school systems turn to computers for manipulation of manage- 
ment information, the introduction of REACT administrative packages 
becomes increasingly feasible; and the people in the school system 
become acquainted with the computer as a tool, the teacher- and student- 
use packages can make an appearance with little instructional change 
in the educational system. 



Context in Which the Project Operates 

Relationship to parent agency . REACT ^s parent agency is the 
Northwest Regional Educational Laboratory (NWREL) in Portland, Oregon, 
a nonprofit corporation supported in part as a regional educational 
laboratory by funds from the United States Office of Education (USOE) . 
NWREL is committed to encouraging local involvement during the devel- 
opment of its outputs. This attitude is based on the conception of 
an increased probability of successful adoption of education innova- 
tions when the adopters are involved in the development of those innova- 
tions. The organizational scheme of the laboratory is indicated in 
Figure 1. 

REACT is a specially funded project which does not depend upon the 
parent agency for any of its budget. REACT functions as a relatively 
small, independent program within the structure of the NWREL, and has 
available to it all existing support services of that agencv. These 
include secretarial and duplication divisions, a media center, and a 
research and evaluation division which provides advice in research and 
evaluation design. REACT has responsibilities to the parent agency for 
meeting formal requirements for filing of evaluation plans, for pre- 
sentation for review of the project's efforts to the board of directors, 
and for general accounting and budgeting. 








3 



FIG. 1. Contextual map. 
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Physical/environmental setting , REACT occupies space within the 
offices of NWREL, sharing a floor with the computer center. Because 
of the computer focus of this project, this is clearly an advantage, 
as there is little or no time and space gap between creation of units 
for computer usage and Inhouse testing of those units in a bench-test 
fashion. Computer-related problems may be spotted immediately. 

The office areas of REACT are open and generally Interconnected, 
with no member working in a room where there is a door to shut. This 
was indicated as facilitating Informal communication, hut with a defi- 
nite expense in terms of desirable working conditions for writing* 

All of the interviewed staff commented on the difficulty of working 
under conditions of constant interruption. However, the top three per- 
sonnel saw the working circumstances as a necessary condition, viewing 
the stream of constant interruptions as basic to their successful per- 
formance of major tasks useful to the project, such as aiaintaining 
contact with cooperating agencies and coordinating the philosophy and 
direction of the two primary training packages. 
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Chapter II: Parameters of the Project 



This chapter contains information about the organizational structure 
of the REACT Project, its staffing patterns, and the roles and functions 
served by its personnel. It also provides description of the outputs 
identified in the study, and illustrates the dependent relationships of 
these outputs in an output map. 



Project Structure 



REACT has two major subdivisions functioning independently in terms 
of activities and decisions, one focusing on administration and the jther 
focusing on teaching. The two coordinators and the Project Director inter- 
act extensively in an informal manner to maintain coordination and agree- 
ment between these separate subdivisions. The two coordinators have co- 
operated on a common set of training manuals to be used in the introductory 
section of both the administrator training oackage and the teacher training 
package. 

Within the administrator training effort resides the development and 
implementation of the MIDAS system — a computerized miniature school system 
with a limited data base which is designed to teach administrators how to 
use the computer to aid them in solving administrative problems. This 
system is used as the basis for developing administrate! training units. 

The coordinator of the administrator training package also cooperates with 
the coordinator of the teacher training package in conducting classes for 
teachers and administrators. These classes are part of the test pattern 
for the two outputs. 

Two distinct activities in addition to the testing and revision of the 
teacher training package lie within the teacher training section. These 
are the development of the newly begun student curriculum units and the 
establishment of the clearinghouse function. The student units are frequently, 
but not always, adaptations of material developed for teacher use, and the 
rationale for the two effortL' is similar. The curriculvm writer employed in 
this division of the project works on development of student packages, and is 
also the coordinator of the clearinghouse at this point in time. The function 
of the clearinghouse is to collect and coordinate for dissemination existing 
Ins true tlor^al units developed for the computer. The clearinghouse effort is 
a logical outgrowth of the work on student packages, since that work involves 
reviewing units which have been developed by other agencies and deciding, upon 
their suitability for adaptation to the REACT system. 

Staff structure . All members of this staff, winh the exception of the 
computer programmer, worked together on a previous project (REACT, Phase 1). 
They know each other very well, and are highly informal in decision and 
management processes. The Project Director and coordinr.tors of the two major 
subdivisions coordinate their thinking about the project on a dally basis. 
Another advantage accruing to this project from the previous phase is in the 
quality of advice available to one curriculum writer from her husband. He 
had been employed on the REACT Project as a mathematician and specialist in 
computer usage. Actions resulting from ♦■he writer's Ccaaultations with her 
husband were cited by several staff nemherc as definitely contributing to 
the project. 
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A disadvantage to th:.s project accruing from the familiarity cimong 
the staff is that there Is no process such as staff meetings to insure 
interaction between all levels of the project. This has the effect of 
insulating the top three people from any opinions or concerns of the 
> three other workers except those that arise in one-to-one discussions 

^ of the work at hand. ‘ There also was little indication in the interviews 

i of interaction between these three workers. 

Figure 2 represents the structure of the project relationships 
existing between the staff members. 




FIG. 2. Project organizational structure. 




-8 . 
i:/^i 



178 



619 



Project rojter * The following staff members were interviewed during 
the onsite visitation or cue interview team. 

Project Director . Assigned .25 full Lime equivalency (FTIi!) to this 
project, he worked on the proposal and was reassigned to ihe project 
for its durat.'.on when it was landed. He was also director of the 
first phase o.^ ctie REACT Project. His responsibilities are adminis- 
trative and supervisory, and he is primarily responsible for final 
reporting of this proiect. 

School Administrator Training Package Coordinato r. Hired during 
previous phase of the project, he was reassigned (1*^ FTE) to the 
project when the current contra^ ‘ was funded. He is responsible for 
the completion of the administrator package and the MIDAS system. He 
shares responsibility for the testing design for the project. 

Teacher Training Package Cooi . dinator . She is supervisor of the 
development of the student curriculum units and the establishment 
of the clearinghouse. She was hired durin-^ the previous pnase of 
the project and assigned to the current project, at 1.0 FTE. She is 
responsible for the completion of the teacher training package, and 
shares responsibility for the testing design of the projeC:. 

Curriculum Writer . She was iiirec (1.0 FTE) for v/riting student 
curriculum units at the project's inception. She worked for a shor*: 
time on the previous phase of the project in a similar capacity. 

Programmed Instruction Specialist . Employed temporarily during th i 
previous phase of the project, he rejoined the current effort at 
its beginu.'.n.g date at 1.0 FTE. He is resp:>nsible for development 
of the writer training process. Thjs process consists of interaction 
between curriculum writers and specialists in instructional technology. 

The following staff member (not interviewed) completes the list of project 
personnel. 

Computer Programmer . He is employed for the duration of the project 
at 1.0 FTE. 



Outputs Generated 

The outputs of work effort that are identified in thxs stud^’ are 
sorted into three categories: products, events, and conditions. A product 

is the tangible result of work effort that results in a surviving trans- 
portable object. An event is an outcome of work effort that results in 
the occurrence of an observable transaction or set of behaviors. A condi- 
tion is the result of work effort that establishes a prerequisite status 
for production or management within the project. 

Index of outputs . The following annotated list describes the outputs 
confirmed by the onsite interview team as major outputs ^or the REACT 
Project. Each output has been arbitrarily assigned an identification 
number. This number is preceded by a letter, P for a product, E for an 
event, and C for a condition. Those outputs interviewed around are 
identified with an asterisk. 'j ^ 






620 



*P~01. Computer-use Instructional Unit for Administrators ♦ 
A written instnictional plan for use in teaching 
school administrators to operate computers as a 
tool to enhance performance of their various 
administrative duties • 

*P-02, Compute r-ut Instructiona l Unit for Teachers . 

A written instructional p n for u5>e in teaching 
teachers to operate computers as a tool to extend 
their classroom teaching effort . 

*P-03 • Computer-use Instructional Unit for Students . A 
written instructional plan for use in teaching 
students to operate computers as a learning 
tool. 



*E-04. Instructional-unit Writer Training . A series of 

inhouse training sessions planned to increase the 
capability of the writers in writing instructional 
units. 



*P-05. 



P-06. 



P-07. 



P-08. 



P-09. 



* P-IO. 



P-11. 



Computer Simulation of School Administration (MIDAS). 

A minature demonstration computer system of hypothet: cal 
school administration. This system includes pupil, 
personnel, financial, facility, and curriculum data files. 

Field Testing Design . The written plan for evaluating 
the various instructional units being produced by the 
REACT Project. 

List of Field Test Sites . A refined list of locations 
which would cooperate in the conduct of field tests 
for the various REACT outputs. 

Clearinghouse Operational Plans . A written operational 
plan for obtaining, classifying, and distributing 
literature about computer applications in school 
situations . 

Student Instructional Units (Application Packages) . 

Written instructional plans to assist students in 
learning subject matter by employing the computer as an 
aid or tool. 

Administrative and Teacher Training Unit (Application 
Package) . A written training manual common to both 
the adminstrator and the teacher. This unit suggests 
ways that administrators and teachers could enhance 
their work by employing computers. 

Revised Field Test (Evaluation Plan) , An ongoing and 
formalized version of P-06. This plan is a requirement 
o f the NWREL . 
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E-12. Assessment of Progress « An ongoing quality control 

effort. The project *s staff considers this to be 
an informal but very essential management tool. 

P-14 . Public Relations P resentation/Demonscration Package . 

A formal demonstration program d(^signed to show the 
possibilities of computer application for students, 
teachers, and administrators . 

P-16. Testing System for Student Packages . A written 

practical set of procedures for testing the studen 
packages . 

P-17. Revised Administrator and Teacher Training Packages 

(Training Manual). A finalized, complete^ and published 
training manual for administrators and teachers. 

P-18. Application Package for MIDAS . / complete training 
package for school administrators to make use of 
the miniature school administration demonstration. 



P-19. List of Products for Clearinghouse A constantly 
growing list of literature about applications of 
computers for extending classroom instruction . 

*P-20. Plan for Future of Project . A proposal for the con- 
tinuation of the REACT Project. 



O 

ERIC 



*C-21. Cap ability in Staff . A condition deemed essential 

by the Director of the project in light of his 
management style . 

*C-22. Managed Money . A management condition deemed essential 
by the Director of the project in the conduct of the 
REACT Project. 

*P-23. Clearinghouse for Educational Applications of 

Computer Technology . The written description of 
the clearinghouse ivity which could be con- 
ducted on a commei 1 basis. 

*E-24. * Modification of Computer Program . The event of 

modifying or changing computer programs to accomplish 
their intended purposes. 

These products, events, and conditions were identified from the 
REACT Project proposal, the project reports, and the interviews conducted 
on site. 

Output map » Figure 2 Is a graphic illustration of products, events, 
and conditions as identified for Project REACT. The figure attempts to 
represent the dependent relationship of one output to another . It should 
be noted that this schematic does not necessarily represent output develop- 
ment in relation to time. It only represents the dependent relationship 
between outputs. 

. .284 
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Chapter Til: Summary of the Data 



The tables included in this chapter each summarize a cate^o>'y of 
data obtained in relation to the various outputs identified for Che 
REACT Project. The column labeled "Project Outputs" identifies the outputs 
of the project which are appropriate to that table. (Only those outputs 
for which data were obtained during the interviews with staff are in- 
cluded in a table. Consequently, the output listing will vary with each 
table.) The numbers accompanying each output (e.g., P-01) are identifiers 
which are constant throughout this project, and can be used in referring to 
the raw data statements relating to that output (see the Appendix). 

The categories of data shown in the tables are the s tandards by 
which the satisfactory completion of the output are ju<iged, the tasks 
required to generate an output meeting those standards, and the enablers 
(knowledges, skills, sensitivities) which facilitate the carrying out of 
those tasks. Within each of these categories is found a list of descrip- 
tive labels which are representative of interviewee statements (raw data). 
In the process of reducing raw data, narrative interviewee statements were 
first linked to one of the above categories, then classified by means of 
a number ^ode under the most representative descriptive label. Each table 
provides the frequency with which interviewees cited specific statements 
(represented by the descriptive labels) of each category. 



Standards Held for Outputs 

Table 1 presents the standards held for the acceptability of each 
output which was a subject of interview. Within this outputs standards 
table there were a total of 30 narrative statements of standards made by 
interviewees. Though these were made in relation to nine project outputs, 
26 bear on primary products of the project. This reflects the current pro- 
ject emphasis on the production of products. The major standards used are 
shown to be "appropriateness of design" and "goal attainment". The fre- 
quency of use of the standard, "acceptance by others," reflects the project 
policy of cross 'becking colleagues* work. The computer-use instructional 
unit for students was subjected to by far the widest rauge and greatest 
number of standards . 



3 

If the reader is interested in the narrative statements of the inter- 
viewees, these can be found in the Appendix. To locate the narrative state- 
ment for any given category, first note the output and its identification 
number in the table. Second, note that each descriptive label within a 
given category has a distinct number or code. Turn to the Appendix and 
locate the output. Under the output locate the category label or heading 
(standard, task, or enabler) and pinpoint the number or numbers (depend- 
ing on frequency cited) of the descriptive label which appeared in the 
table. The statement in the Aj?pendix opposite this number is the original 
narrative statement from an interviewee and is only represented in the table 
by the descriptive label and its number coding 
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Output Standards Cited for Each Output Analysed 






Idb.le 2 presents the standarJs I’sed for judging the acceptability 
of management activities related to project outputs. (Process standards — 
the acceptability of the performance of processes and operations.) 

Twelve process standards were cited in narrative interviewee state- 
ments. The majority of these applied to having a condition of capability 
among the staff members. The major standards applied to judging capability 
in staff are showri to be: ^Values and objectives match’* and ’’acceptable 

level of output.” 



Tasks Pertaining to Output Attainment 

Table 3 presents the tasks performed in the accomplishment of those 
project outputs that were the subject of interviews. Interviewees made 
159 narrative task .statements. Of these, the tasks cited primarily are 
’’producing kinds” of tasks, 90 of the 159 cited. ’’Managing kinds” of 
efforts were cited 69 times. 

The table indicates that the current major effort is in production 
with a major concern in the area of operating in an accountable manner. 
The primary products currently being emphasized are the computer-use 
instructional units. The major management concern is a condition of 
high capability in staff. 



Enablers Pertaining to Output Attainment 



There were 30 narrative statements of required knowledge made by 
REACT interviewees. Of particular note in Table 4 is the requirement 
for knowledge of technical subjects (primarily computers, in this case) 
and a knowledge of the state of computer applications in education as 
evidenced by the items coded to ’’Project variables, exteriial.” 

In Table 5, which lists the skills required for accomplishing the 
project outputs, narrative statements of interviewees did not indicate 
major emphasis in any particular skill area. This is somewhat surprising 
when considered in light of the previous table showing a need for tech- 
nical knowledge. However, the broad range of skills cited and their 
almost equal emphasis should be noted. 

Within the 41 sensitivities (Table 6) cited by interviewees, as 
being required for the accomplishment of project outputs, the only em- 
phasfts appear to be a sensitivity to the ’’needs of self /others” and a 
sensitivity to ’’existing value systems.” The interviews showed the re- 
spondents to be quite aware oi the possible reluctance of school systems 
to change their current modes of teaching and administering to a mode em- 
ploying compu ters . However, the s taf f was enthusiastic in their stated 
belief that their employment of computers could meet many of the needs 
of the target audience. 
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04 Deadlines are met 

05 Acceptable level of output 
13 Work conducted w/in budget 

21 Follow-on proposals are funded 
26 Values and objectives match 
31 Overtime worked voluntarily 
34 Impact of effort, favorable 
42 Revision yields improvement 
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Chapter IV: Supplementary Data 



Included in this chapter are aata about output classification and 
interrelationships withiv the project, the backgrounds of project per- 
sonnel, and the support resources of the project. 



Clr. ssif icatxons of Output Characteristics 

Outputs may be categorized in terms of a number of variables. Amoii 5 
them are structure (product, event, or condition), (b^) Function 
(policy setting, management, or production), (c) Level (focal, component, 
or facilitating) ► (<d) Character (knowledge, technology, implementation, or 
information), and (je) Stage of completion. These five schema are repre- 
sented in Table 7 for each project output idr^ntified, with frequencies 
summarized for each category. 



Summary of Staff Backgrounds 

Personnel data were obtained from four of the six professional staff. 
Two have doctoral degrees and two have bachelor's degrees The Director 
has been associated with the project (including the preceding Phase 1) 
for 25 months, with the remainder of the staff varying from 9 to 17 months. 

Of the four responding professional staff members, one indicated 
seven years of previous educational research and development experience, 
one indicated four years, one indicated two years, and one indicated no 
previous educational R & D experience. However, this last respondent 
indicated seven years of R & D experience in fields other than education. 
Prior experience in educational a'^’ministration was listed at 10 years 
for one, four years for two, and none for the fourth staff member. Agai.i, 
however, this last respondent indicated 14 years of administrative experi- 
ence outside of the educational realm. 

The Director indicated that he considered his .oiowledge of "how to 
work with people" as a primary asset. His most important skills were 
cited as "personnel assignment" and "contract negotiating." Primary know- 
ledges listed by other staff members included: "general knowledge of public 

schools," "computer technology," "curriculum development techniques," and 
"instructional technology as developed in industry." All respondents indi- 
cated that interpersonal skills were mandatory, as were writing and speak- 
ing skills. Two of the respondents indicated a need for being sensitive to 
the problems of teachers, administrators, and students. 
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P-11 tavlaad Fla Id Taac lEval.) Plan 


X 


X 


X 


X 


X 


E-12 Aaaaaaaant of ProgrMa 


X 


X 


X 




X 


P-14 P.g. Praaancaclon Dnaonacraclon pkg* 


X 


X 


X 


X 


X 


P-16 Taaclng Syataa for Studant Fkga. 


X 


X 


X 


X 


X 


P-17 lavlaad AdalnlaCraCor and Taachar 


X 


X 


X 


X 


X 


Training Pkga. (Training Manual) 












P-lB AppllcaClon Package for MIDAS 


X 


X 


X 


X 


X 


P-lf Llac of Producca for Claarlngbouaa 


X 


X 


X 


X 


X 


P-20 Plan for PuCurn of Project 


X 


X 


X 


X 


X 


•C-21 Capnblllcy in Staff 


X 


X 


X 




X 


•C-22 Managed Honey 


X 


X 


z 




X 


•P-23 Claarlnghouae for Educational 


X 


X 


X 


X 


X 


Appllcaclona of Co«puCar Teclmology 












*E*24 Modification of CoaipuCar Prograa 


X 


X 


X 




X 


Claaalflcaclon Prequanclaa 


17 3 2 


1 10 11 


7 5 10 


0 17 0 0 


0 4 1 11 2 4 



The apaclflc output 


charactatlatlca are 


Identified aa followa: 






Structure 


Functlo'i 


Laval 


Character 


Conplaclon Staae 


p - product 
a - ayanc 
c - condition 


pa “ policy aacclng 
■ - nar*agcBeat 

p - product 1 m 


fl - focal 
c - conponant 
f2 * facilitating 


k - knowledge 
c - technology 
ll - laplanantacion 
Ig - Information 


1 - completed over one year ago 

2 - conplacad 3 to 12 anneha ago 

3 - complaced within laat 3 noa* 

4 - currently In prograaa 
3 - not yet underway 
4-00 going (contlnuoua) 
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Support Resources and Equipment 

Thjse support resources used by the personnel interviewed and seen 
by them as essential to the project are as follows: 

Typing. 

Stenographic services . 

Computer programming. 

Graphics. 

Printing. 

Library Research. 

The support equipment required by the project staff included: 

Dictating equipment. 

Calculators. 

Key-punch machines. 

Remote computer terminal. 

Onsite computer. 

Photographic equipment . 

Card sort machine. 

Video tape equipment. 

Duplication machines. 
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Chapter V: Project Dynamics 



RKACT io a small » nearly autonomous organization working within 
the overall context of NWREL. Its professional staff of six has been 
carefully selected to provide a highly qualified, compatible working 
team. The purpose of this chapter is to show the manner In which this 
team performs. 



Management and Communlc< >:ion Processes 

The Project Director manages the REACT operation in a rather informal 
style. His functions in the project prevent him from constantly ov^^rseeing 
staff work, his stated expertise lies (to a great extent) in the areas 
of staff selection and assignment and in contract negotiating. His work 
requires frequent travel and constant interpersonal relations with and for 
the parent agency and with outside agencies. In filling this type of role, 
he becomes aware of personnel and their qualifications throughout various 
areas of expertise in which he is interested. He further makes a point of 
being aware of promising students in various schools throughout the country 
which have an earned reputation of producing highly qualified graduates. 

The knowledge he thus gains allows him to obtain staff members with 
the expertise he requires. The Director emphasized, however, that such 
staff selection and hiring can only be done within the bouiids of profes- 
sional ethics. 

Because of his management style, uhe Director requires staff that 
not only are well qualified in professional and technical areas, but 
are capable of accepting the responsibility of being the final approving 
authorities for work done by themselves or under their direction. 

The Director's management style is one of delegating both responsi- 
bility and authority, retaining for himself only the authority to act 
as arbitrator in settling disputes. In this way he can act as a team 
member, bringing his expertise for negotiation into play to provide 
facilities, equipment, and working conditions which will enhance the 
operation. 

He operates with a minimum of policy restrictions placed upon his 
staff. For example, he does not rigidly enforce set working hours. 

He expresses the belief that conscientious people who arc dedicated to their 
work will frequently put in more than their required time to complete work 
if given some freedom in their working hours. In some cases this freedom 
may conflict with agency policies. In a case of this sort, he acts as 
buffer or go-between for his staff. 

The Director keeps himself well aware of the '>roject status by 
maintaining an open-door policy. The entire staff works on a first-name 
basis and reports their progress almost daily. 
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LD^SLL^i* ionb li ips Among Projo c t_ 

Each staff member has Ills own area of expertise. Since each was 
hired on the basis of that expertise there is an obvious air of respect 
evidenced between them. Though no stated policy to this effect was 
found, there appears to be a custom among the staff members of asking each 
other to critique their work. This action serves a coordination function 
as well as providing input from various points of view. 

Though staff meetings are occasionally called by the director to 
discuss major efforts requiring input from all staff, a preferred method 
is a frequent informal "get-together" over lunch. 

Two or more staff meinbcrs frequently will voluntarily combine their 
skill and knowledge to accomplish a required task. For example, a pre- 
sentation to the board of directors of the NWEL was scheduled. Two of 
the staff cooperated to prepare and present a demonstration on the current 
status of the REACT Project. The logistic and presentation problems were 
handled as a cooperative venture. This particular presentation was made 
during the Director's absence. 

I nterrelationships Between Project and Parent Agency 

Though a project of NWREL, the REACT Project operates in a rather 
autonomous fashion. The project staff (with the exception of the Director) 
is committed full time to this project. This autonomy of operation may, 
in part, be due to the fact that the project is specially funded and 
does not depend upon the Laboratory for any of its budget. The REACT 
Project, however, is conducted as a NWREL project and as such complies 
with NWREL policies. 

The future plans for the REACT Project include the use of the mar- 
keting facilities of the Laboratory . Project staff members cooperate 
in the preparation and dissemination of promotional materials. 
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Chapter VI; Impllcatlonn for Training 



Doctoral level training was not seen by the staff of this project 
as having any other than political value, but that element was seen as 
essential. The preparation most often mentioned as actually useful to 
the performance of the tasks of this project was general experience In 
the fields of public school education and administration, and computer 
technology. The more varied and responsible the experience, the more 
utility it was seen as providing. The implication drawn from this 
data is that real life (or simulated) experience in normal working 
conditions jhould be a major integrating component of a training [)rograni 
for persons intending to do work similar in nature to that Involved In 
this project. 

Experience (real or simulated) as mentioned abov?, coupled with some 
specific items cited by individual staff members, could be combined Into 
project simulation. One staff member noted that he felt a deficiency in 
the area of knowledge of funding sources. Another ident.'^fied proposal 
writing. Still another stated that technical details of computer use was 
an area in which he nad to do considerable outside study . Putting these 
items together into a single simulated situation appears to be a distinct 
possibility. For example small, short-term developmental projects could 
be attacked as a small class effort. The proposal could be cooperatively 
written, with funding obtained anc practical uork experience gained as a 
functioning team. 
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Appendix: Listing of Output Standards, Tasks, and Enablers 



The following is a list of standards, tasks, and enablers for out- 
puts around which interviews were conducted. These statements were ex- 
tracted from discussions with interviewees and were coded into their re- 
spective category sets. The selected code precedes the statement and 
indicates the following for: 



STANDARDS 

Code J: Structure of Standards. 

J~1 Standards against which outputs are Judged. (output oriented) 

J“2 Standards against which processes and/or operations are judged, 

(process oriented) 

Code LM: Primary categories for Standards. 

TASKS 

Code NO: Clusters of Tasks. 

ENABLERS 

Code S: Structure of Enablers. 

S-1 Knowledge. 

S-2 Skill or ability to perform. 

S-3 Sensitivity or awareness. 

Code UV: Primary Categories of Enablers (knowledges, skills, or 
sensitivities) . 

The codes associated with these three categories (standards, tasks, 
enablers) are the same both here in che listing and as previously cited 
in Chapter III tables. 

Each of the analyzed outputs is cited below wltnln a rectangular box. 
Listed under each are the interview statements rele/ant to that output. 



E-04: Instructional-unit Writer Training 




STANDARDS; 

J 

2 05 The writer or trainee produces an instructional instru- 

ment that reliably teaches, as evidenced by the test scores 
in a field test of the Instrument. 
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tasks: 



m 

29 Have trainci write a short instructional sequence 

based on a concept. 

24 Read short instructional sequence for clarity in 

order o;: presentation. 

26 Discuss with trainee what parts of instructional 

sequence he has written that are not acceptable. 

29 Have trainee rewrite instructional sequence using 

hints prescribed. 

26 Repeat discussions and rewriting of the short 

instructional sequence until it is Judged acceptable. 

05 Administer trainee * s short instructional sequence 

to one or two naive persons in the field. 

05 Evaluate ability of short instructional sequence to 

teach by testing the field subjects using the 
test develped by the subject area expert. 

24 Modify the short instructional sequence from ex- 

perience with field subjects till it is effective 
(instructs) as measured by the test. 

05 Adminis ter modified instructional sequence to 5 

to 10 new subjects to validate its teaching ability. 

29 Have trainee repeat instructional program writing 

training process many times on different subjects 
till trainee reaches an acceptable level of writing 
skill. 



enablers: 

s uv 

1 04 Know techniques of behavior change or behavior 

shaping. 

1 04 Knowledge of the area of applied instructional 

technology, 

2 06 Skill in the application of program learning 

techniques to instruction. 

2 24 Skill in writing behavioral objectives. 

2 10 Skill in analyzing the effectiveness of items 

in an instructional sequence against a test or 
evaluation measure. 



P-02: Computer-use Instructional Unit for Teachers 



STA^^DARD^;: 



J 

T 05 
1 13 
1 30 

TASKS : 

m 

01 

03 

03 

04 
06 

04 

05 

06 
24 
22 



ENABLERS : 

1 UV 
1 04 
1 06 

1 05 
1 06 
1 01 
1 06 
1 06 

1 04 

2 01 
2 06 
2 IS 

2 06 
2 14 
2 29 

2 37 

2 17 
2 23 
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Usnbility of units in teras of project phixosophy. 
Consensus of writer and supervisor. 

No negative f eedback. 



Def ine philosophy for compute r-extended instruction . 

Select areas within a curriculu.n appropriate for 
computer applications. 

Define criteria for developing of a computer application 
in an instructional subject area. 

Write a segment of the computer-use instructional unit. 
Analyze writing ag'^inst criteria for computer applications 
of an Insti actional subject area. 

Write training Mnit Introducting teachers to the computer. 
Tryout introductory teacher training unit with available 
teachers in the school dist rict . 

Revise teacher training materials on the basis of tryout 
results. 

Establish procedure for final revisions of training 
instructional units . 

Monitor feedback on utilization of teacher training 
instructional units. 



Knowledge of computer limitations and costs. 

Knowledge of classroom computer applications in 
existence and their functionality. 

Knowledge of the recognized experts in computer application. 
Exposure to teacher use of computers. 

Know the instructional objectives teachers seek. 

Knowledge of current thinking in curriculum methodologies. 
Knowledge of good and bad uses of computer. 

Knowledge of how to use the computer in extending 
instruction. 

Must be able to teach or have taught. 

Computer programming skill. 

Ability to use State recommended documents on 
curriculum for developmental purposes. 

Writing materials for computer-aided instruction. 

Writing at target level. 

Sense of timeliness in directing closure on a 
writing task (skill) . 

Ability to select writing staff which can interact 
and work within context of critical interaction. 

Ability to translate and define computer related jargon. 
Skill in developing sensitivity and acceptance in 
others, re. limitations of computers. 
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s w 

2 32 Skill Ir. Identifying Indicators of teacher training 

program weaknesses as shown by criteria tests. 

2 18 Skill Jn determining the relevance of test items to 

objcctl ves . 

3 09 Awareness of various alternatives In Instructing. 

3 23 Sense of cost-benefits In linking computers to objectives. 

3 52 Sensitivity to Impact of crltlc;i3m on staff writers. 

3 47 Have tact but be willing to confront people qualitatively. 

3 08 Sensitivity to verifying behavioral objectives as an 

appropriate activity. 

3 03 Understanding the needs of the user as demonstrated 

by feedback on objectives, criteria tests, and 
critiques. 



P-03: Computer-use Instructional Unit for Student 



STANDARDS : 

J 

1 11 The computer Is used In a way that does Justice to its 

capability, allowing freedom to student. 

1 11 The item Is not something that could be equally well 

taught by teacher without use of computer. 

1 09 Accuracy to subject. 

1 04 Flow of writing enables easy aiderstanding. 

1 11 Material is appropriate for audience for whom written. 

1 09 No omissions in coverage of Item. 

1 12 It develops conceptual understanding rather than mere 

compu t a t ion al ab 11 it y • 

1 11 The simulation Is justified by doing something that 

cannot be carried out more effectively in real life. 

1 12 Allows latitude for student's Individual creation. 

1 01 Agrees with preliminary format as to what each 

section of the unit should contain. 

1 13 Other staff critiqued item as satisfactory. 

1 07 Feel right about final written copy. 

1 28 Successfully used by unfamiliar teacher in bench test. 

TASKS : 

NO 

04 Write student curriculum units in mathematics, which 

will integrate In the computer the kinds of things found 
in Algebra I, Geometry, Algebra II. 



01 

02 



01 

01 

04 

03 

04 
04 
04 
04 
04 

04 

05 

05 

04 

02 

06 
06 

03 
02 
33 

33 

22 

25 

04 

05 



ENABLERS: 

S UV 
1 01 
2 01 
2 22 

3 23 

3 01 
3 50 
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Examine existing matcrl^ils for usability or adaptability. 
Discriminate against computational skill develDpmcnt 
kind of unit (as being done already) > and look for 
units to develop student concepts. 

Look through material to avoid repeating what has 
already been done. 

Examine new text books to determine what e lemen ts are 
suitable for use with computer. 

Make outlines of topics contained In textbooks to 
generate composite subject outline. 

Make list of ideas for suitable units for computer 
treatment. 

Write unit according to previously written format. 

Write Introduction outlining purpose of unit. 

Sketch basic concepts needed before going to computer. 

Write explanation of concepts needed. 

Explain concept of **llnear. " 

Write next section which Is warm-up exercise to check 
understand Ing of prerequisite concepts . 

Go through lesson at teletype--gulde through what to do. 

Guide student to make ’’discoveries’* intended foi him 
(what to do at teletype end afterwards, with data). 

Write section on application of this concept (discovery). 

Face problem of coverage In one lesson relative to 
constraint Imposed by previously written format. 

Decide to break treatment down to separate lessons at 
the teletype, rather than trying to do it all at once. 

Rewrite section, getting student working on line equation 
at teletype . 

Consider format with other staff. 

Discuss notion that hardware programming should be included 
Decide to supply programs in three or six different computer 
languages. 

Decide to Integrate study material with computer--use sessions In 
smallei segments . 

Set own timelines for getting work done. 

Decide to drive for lesser number of quality 
units, rather than greater number of Inferior units. 

Write format for "quickie" type unit to help students 
use computer as a "sophisticated adding machine." 

Bench test units, to watch for weak spots in student 
handling. 



Subject (mathematics) . 

Teaching of subject for 3 1/2 years. 

In evaluating work without reference to whether it is 
way I would have done it. 

To what computers can and ought to do in the learning 
process. 

Sensitivity to own prejudices. 

To ask self: "Is it appropriate?" "Have I made best use 

of computer?" 

To feedback from pilot use of material. 

2G2 



3 22 






3 


03 


To 






or 


3 


27 


To 



neoti for time to allow students to acquire 
realization of principle to be conveyed, 
need for keeping to small pieces. 
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C-21: Capability in Staff 



STANDARDS : 

J 

2 04 Timelines are met. 

2 05 High quality work is produced. 

2 26 Staff is conscientious. 

2 31 Staff members willingly work overtime to achieve results. 

2 26 Staff is loyal. 

2 05 Staff is productive. 

TASKS : 

m 

21 Determine capabilities required by job. 

29 Maintain contacts with schools that are known to produce 

qualified people. 

29 Frequently visit sites that do similar work. 

01 Observe people working at their regular jobs. 

21 Discuss project problems with people working in other organizations. 

21 Consider the qualifications of outside people. 

21 Mentally rank people in field as to their qualifications. 

23 Hire selected people for occasional consulting jobs. 

21 Consider how selected capable people in field (as personalities) 

would fit in project organization. 

21 Contact potential candidates to make kiiovn vacancy. 

21 Discuss requirements of position with potential candidate. 

21 Describe working conditions (job security) with desired person. 

21 Offer more money than expected to offset insecurity (set by 

project duration). 

21 Hire staff member for project. 

22 Assign responsibility to staff member. 

2/ Relinquish to staff member the authority to make possible 

the accomplishment of the responsibility. 

26 Advise new employee administratively if required. 

26 Advise new employee technically if required. 

25 Concur with staff members* decisions. 

25 Resolve differences of opinion between staff members. 

25 Establish policies when required. 

24 Monitor staff efforts. 

31 Meet with staff frequently and Informally. 

25 Act as buffer between staff and agency. 



ENABLERS : 



/ 



s vv 

1 12 Know schools which produce consistently good in areas of interest. 

1 12 Know agencies employing people with expeitise you need. 

2 02 Skill in interacting with people. 

2 26 Maintaining knowledge of field (c::pertise) . 

2 26 Maintaining knowledge of people (keeping track of it). 

2 37 Judging capabilities of people. 

2 48 Skill in getting people to like you. 

3 02 Sensitive to sources of dissatisfaction of potential 

candidate in his present position. 

3 03 Sensitive to potential candidate’s need for supervision. 

3 03 Sensitive to needs of potential candidate. 

3 37 Be willing to relinquish authority. 

3 38 Be sensitive to "tone" of project work (is more developing 

in system?). 

3 58 Be willing to support staff’s decisions. 



C-22: Managed Honey 



STANDARDS : 

J IM 

2 13 Categories not overspent. 

2 13 Adequate production accomplished without available money. 

1 13 Budget approvea by superior (agency). 

TASKS : 

NO 

22 Establish a spending plan. 

22 Categorize planned spending in a budget. 

31 Present budget for approval* 

22 Justify spending plan as required. 

22 Establish an accounting ystem for staff. 

22 Apportion operating funds to staff. 

22 Assume legality of expenditures* 

22 Retain final approval authority as a monitoring function. 

22 Advise staff on major expenditures* 

22 Monitor adherence to accounting proceduring* 

ENABLERS : 

s m 

1 11 Know monetary requirements of project. 

1 11 Know current status of spending (budget). 

O 

ERIC 
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1 13 Know sources of funds# 

2 22 Bo. able to project cost estimates over a year of operation. 

2 38 Be able to get value for expenditure. 

2 38 Be able to spend where it will do most good. 

3 IZ Be sensitive to monetary needs as new problems develop. 

3 23 Sensitive to the value of what you get for your money. 

3 37 Be willing to allow staff to spend. 

3 40 Sensitive to the need to account for money. 



P-01: Computer-use Instructional Unit for Administrators 



STANDARDS ; 

J 

1 07 Meets my own personal standards. 

1 12 Field test results show materials to be satisfactory (achieve 

desired results). 

1 06 Users buy (or accept) instructional unit. 

TASKS : 

NC 

01 Study literature to gain knowledge of status of computer 

use in school administration. 

01 Collect detailed reports from other projects in field. 

01 Study reports from other similar or relevant projects. 

01 Select appropriate ideas from other similar projects. 

02 Conceptualize unit of instruction in light of needs of public 

school administrators. 

04 Outline manual for administrative unit. 

02 Confer with school admiiiistrators concerning unit content. 

04 Write draft of manual using outline as guide. 

06 Study draft of manual to identify poorly presented areas. 

06 Edit draft to improve format and presentation. 

22 Direct secretary to type final copy. 

06 Proof read final copy. 

03 Plan details of field-test presentation of course (instruc- 
tional unit). 

31 Confer with consultant as to evaluation of instructional unit. 

03 Prepare formal evaluation plan for agency records. 

04 Construct measuring instrument (questionnaire). 

23 Select field-test site for test of unit. 

29 Confer with personnel at field-test site as to procedures 

to be followed. 

Plan detailed arrangements for evaluation with site personnel. 



^.0 
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03 Plan presentation of course (unit) at field-test site, 

05 Teach administration unit for field-test, 

05 Collect field-test data by questionnaire, 

05 Interpret field-test data to determine effectiveness of course, 

06 Revise administrative unit (if necessary) in light of evaluation 
results , 

06 Rewrite unit if necessary, 

04 Print final version of unit, 

ENABLERS : 

5 W 

1 04 Have basic knowledge of computer technology. 

1 04 Have basic knowledge of public school administration. 

2 14 Able to write clearly and concisely. 

2 15 Able to speak to groups (oral presentation) , 

2 35 Expressing your thoughts in terms meaningful to vschool administrators. 

2 37 Able to judge the writing of others. 

2 49 Able to interact productively with individuals, 

2 49 Able to interact productively with groups. 

3 02 Sensitive to administrators’ problems and needs, 

3 03 Sensitive to other people's motivation, 

3 16 Sensitive to agency's policies, 

3 20 Sensitive to agency's image. 



P-05: Computer Simulation of School Administration (MIDAS) 



STANDARDS : 

J IM 

1 12 Product illustrates standard (common) needs of administrators. 

1 12 Target (administrators) obviously understands the illustrated 

applications • 

2 34 Target (administrators) visualize application to their own 

situations • 

TASKS : 

01 Review relevant literature • 

02 Conceptualize system in relation to needs of school administrators. 

02 Confer with school administrators to confirm their basic needs. 

03 List items to be included in data files* 

03 Conceptualize interrelationships of data files; conceptualize 

computer retrieval system. 

03 Write data (hypothetical) for inclusion in data files. 
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31 Confer (direct) with programmer in reference to programming 
required 

24 Confer with programmer to maintain awareness of status of work 

(monitoring) . 

32 Confer with consultant concerning promotion for MIDAS system. 

07 Prepare (write) promotional materials for diffusion of MIDAS system. 

31 Present demonstration of MIDAS at agency's Board of Directors 

meeting. 

31 Discuss potential applications of MIDAS at agency's Board of 

Directors meeting. 

ENABLERS : 

S W 

1 05 Know information needs of school administrators. 

1 06 Know similar type computer systems in actual use. 

2 11 Be able to accept responsibility of being approval authority 

for own work. 

2 20 Be able to be realistic in building content of data files. 

3 02 Sensitive to school administrators' limitations of function. 

3 09 Sensitive to using an approach that will appeal to public school 

administrators • 

3 16 Sensitive co desired agency policies. 

3 20 Sensitive to desired agency image. 

3 33 Sensitive to the need for quality work. 

3 53 Be able to be creative in conceptualizing system. 



P-IO: Administrative and Teacher Training Unit (Application Package) 



STANDARDS : 

J 

1 07 Meets my own personal standards. 

1 12 Field-test results show materials to achieve desired results. 

1 06 Users purchase (or accept) materials. 

TASKS : 

NO 

01 Study literature to gain knowledge of current status of computer 
use by teachers and administrators. 

02 Confer with colleague to determine elements that are common (or 
basic) to computer use by both teachers and (scope of manual) 
administration. 

04 Outline manual of instruction for areas of computer use common 

to teachers and administrators. 
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02 Consider objectives that outline suggests. 

02 Involve potential users (teachers and administrators) in planning 

by conferring on outline of unit. 

02 State objectives in terms meaningful to users. 

04 Draft manual using outline and objectives as guides. 

06 Study draft, to identify areas needing improvement. 

29 Interact w: th colleague to insure compatibility with teacher unit. 

06 Make corrections and adjustments as advisable. 

22 Direct secretary to type final revision. 

06 Proof read final copy . 

ENABLERS : 

s m 

1 04 Know basic comp iter technology. 

1 06 Current status ^f use of computers in education. 

2 14 Able to write c early and concisely. 

2 15 Able to make formal presentation. 

2 35 Able to express your thoughts in terms meaningful or acceptable 

to both teachers and administrators. 

2 49 Able to interact productively with colleagues. 

3 02 Sensitive to problems and constraints of teachers and administrators. 

3 03 Sensitive to needs of teachers and administrators. 

3 03 Sensitive to other people's motivation. 

3 16 Sensitive to agency's policies. 

3 20 Sensitive to agency ' s image , 



P-20: Plan for Future of Project 



STANDARDS : 

J m 

1 14 Approval by funding agency. 

2 21 Receipt of funds. 

1 13 Parent agency approval is gained. 

TASKS : 

m 

24 Judge impact project is currently having by contacts with field. 

32 Discuss current project with field contacts* 

02 Discuss desired focus of future effort with review panel. 

02 Discuss proposed plans with field contacts* 

01 Visit potential field sites. 

02 Discuss plans with potential field sites. 

21 Identify field workers desired at potential field sites. 
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23 Discuss possible consortium effort with potential field sites, 

23 Select field sites for inclusion in proposal. 

04 Wri,.e description of effort to be made (component of project). 

22 Project cost estimates for portions of proposal. 

04 Prepare summary budget for proposal. 

04 Write draft proposal for future effort. 

22 Direct that draft of proposal be typed. 

06 Proofread draft of proposal. 

22 Direct that final typing be done. 

04 Duplicate proposal as required by potential funding agency. 

ENABLERS : 



S UV 

1 06 Know current status of field (computers in education) in general. 

1 12 Have personal knowledge of who is expert in field. 

1 12 Have personal knowledge of people’s capability in field. 

1 27 Have direct personal knowledge of potential field sites. 

2 37 Able to judge capabilities of people. 

3 16 Senstivity to political attitude in potential field sites. 

3 16 Sensitive to reputation of potential field sites. 

3 16 Sensitive to frame of reference of potential funding agencies. 

3 57 Sensitive to imminence of deadlines. 



E-24: Modification of Computer Program 




STANDARDS: 



J IM 

1 22 The modified program successfully performed in the computer. 

2 42 The modified program was shorter and more efficient 

than the original, in that file manipulation, chaining, and 
inaction with other computer programs was reduced. 

TASKS : 



NO 

01 

06 



06 



Review original program to determine where modification 
can be made and still provide the desired output. 
Rewrite program to remove long literal strings from 
the continuous operation of the program but that could 
be called upon when they were needed. 

Convert modified program to paper tape using teletype 
console. 

Feed modified program into computer from the paper tape 
using the teletype console. 
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ENABLERS : 

S UV 

1 04 Know BiSTC language (computer). 

1 22 Have to know, of the two computer systems available, 

which one is most efficient time-wise and which one is 
most efficient cost-wise. 

1 22 Have to know the monitor commands to the computer to 

delete portions of the program being modified for the 
conversion from one computer to another. 

1 22 Know the differences in BASIC languages (computer) between 

the two computers working with. 

2 27 Able to operate teletype to prepare paper tape for input to 

computer. 



P-23: Clearinghouse for Educational Application of Computer Technology 



STANDARDS : 



J 

1 11 Materials relevant to level (e.g., school, rather than 

college) . 

1 11 Materials relevant to type of computer use specified for 

this project. 

TASKS : 



m 

01 

01 

04 
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Gather all materials in computer-aided instruction 
relevant to the function seen for computers in this 
project. 

Evaluate available materials relative to computer^use 
functions . 

Wrice a report (suggested quarterly) giving availability, 
evaluation of material, how to acquire, kinds of 
course appropriate for the background necessary to use 
it, time taken in use, by subject area (English, math, 
science, etc. ) . 

Draw up a letter and send to 75 or 80 projects to 
obtain participation. 

Receive and examine responses to letters requesting 
participation. 

Design regular format for project descriptions. 
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KNABLEKS ; 



Examine project responsea to determine context befo 
filing away for future referencing. 



No Informstion collected under this heading. 
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This profile has been prepared according to 
PROFILE FORMAT No. 2 

Three profile formats are represented in this volume. 
The leader should refer to this number when making 
use of the reader's GUIDE to the profiles. 
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Chapter I : Overview 



This chapter contains a narrative introduction to the Integrated 
Career Development Curriculum Project, including the objectives, rationale, 
and significance of the project and the context in which the project 
operates . 



Synopsis of the Project 



Title : The Integrated Career Development Curriculum . 

Responsible Agency: Western States Small Schools Project. 

Funding Source: U.S. Office of Education. 

Funding Duration: July 27, 1968 to December 31, 1972. (4 years and 

5 months) 

Observation Dates: April, May 1971. 

Present Stage of Development: Middle stage, which is the curriculum 

development phase or set nd phase of 
the project. 

RDD&E Focus of Project: To date, primarily development (the next 

phase will be a testing effort). 

Expected Outcome: Single Concept Learning Units Package (SCLU). 

Level of Funding and Duration: Medium-High. (level 5 of 7 levels) 

Agency Setting : Public education . 

Setting of Primary Location of Work Efforts: Consortium of Utah, 

Nevada, New Mexico, Arizona, and Colorado School Districts. 

Staff Summary (Current): Professional Supp '^t 

Total Full Time Equivalence (in man 

years) : 3.00 2.00 

Number of Personnel Assigned: 7 2 

Professional Specialities of Staff (interviewees only): educational 

administration, guidance/ counseling , educational psychology . 



O 
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Oblectlves, Rationale, and SiRniflcnnce of the Proleci 

The Inte^i^rated Career Development Curriculum Project (ICDC) is an 
outgrowth of an extensive effort by the Western States Small Schools 
Project (WSSSP) to find ways In which rural youth can better prepare 
themselves for careers in both rural and urban settings. During the 
period 1965-196b, WSSSP created a Career Selection Education Program, the 
forerunner of the ICDC Project. It was recognized that a genuine cur- 
riculum which would serve the students in making decisions about career 
opportunities and communities in wliich to live was lacking in the original 
program. As a result of this finding, the ICDC Project was initially 
funded in the amount of $303,300 for the period July 27, 1968 to June 27, 
1971. 



The overall objective of the ICDC Project is to facilitate career 
preparation by young people attending isolated small schools but who 
ultimately may become employed in more complex urban communities. To 
provide the means whereby students are helped to understand their own 
capabilities and the career opportunities commensurate with those abilities 
and talents, a curriculum based upon the general capabilities necessary 
for most occupations is being created. 

A panel of consultants, the Quality Assurance Panel, is assisting 
in the development of this curriculum. The Panel is made up of experts 
in the fields of rural sociology, vocational education, rural education, 
and educational psychology, and is asked to make recommendations and de^ 
cisions about every important operation of the project, i.e., performance 
objectives, single concept learning unit prototypes, production procedures, 
curriculum strategies, and instructional strategies. 

Supporting the major objective, which is to develop and evaluate an 
integrated career development curriculum appropriate for Ui;e in isolated 
small high schools located in sparsely settled areas, are the objectives 
of each of the project’s three phases. These objectives are: 

Phase I - (a) to define the concepts, skills, and 
attitudes students should possess 
on completion of the curriculum, and 
(Jb) to develop terminal behavioral ob- 
jectives for each component of the 
integrated career development cur- 
riculum. 

These Phase I objectives were carried out by a 
curriculum development task force of curriculum, guidance, 
and vocational education specialists. Along with the 
work of this task force, an outline for the Integrated 
Career Development Curriculum as a whole was developed 
through research conducted by a panel of consultants. 

This outline defines the gross educational objectives 
of a Career Development Education program for small 
schools and furnishes the principal components of the 
proposed curriculum being develop' . 

Phase II - (^) to develop the proposed curriculum and 
appropriate instructional materials in- 
cluding course outlines, instructor 

'■ 21V 



handbooks , student study materials , 
training aids, and evaluation instru- 
ments; and 

to conduct a preliminary try-out and 
evaluation of the developed curriculum 
and materials. 

Phase II objectives are being carried out by the cur- 
riculum task force which, with the assistance of subject 
matter specialists, is selecting/developing instructional 
materials and devices. 

Phase III - (a) to inform all staff members of the 14 
pilot schools about the :^urriculum 
and their roles in the activity, 

(lb ) to train a ^'career selection agent** to 
serve as a project coordinator and 
staff trainer for each of the 14 
cooperating high schools , 

(Oto try out and evaluate the developed 
curriculum and instructional materials 
in 14 cooperating pilot schools, and 
(jd) to prepare and disseminate the results 
of the study. 

These Phase III objectives will be attained in part 
through a one-week workshop to familiarize all faculty members 
of the cooperating pilot schools with the curriculum and in- 
structional materials, and a four-week institute to train 
a **career selection agent" to serve as a coordinator and 
teacher trainer for each of the 14 cooperating pilot schools. 



During the period of 1963-1964, when WSSSP was working in the areas 
of enlarging the curriculum, individualizing instruction , adapting tech- 
nology to the classroom, .nd experimenting with flexible scheduling, the 
project personnel became interested in getting students aware of a wider 
range of vocational opportunities . A pilot study which used the extended 
community as a laboratory was begun at Meeker High School in Colorado. 

This experiment was called the Vocational Exploration Program, involving 
business people, professional people, and the high school students of 
the community. 

Other small rural high schools in the five states of Utah, Nevau 
New Mexico, Colorado, and Arizona began similar programs during 1965-1968 
with the backing of WSSSP. These developments led to the Career Selection 
Education Program which in turn led to the ICDC Project and the conception 
of an integrated career development curriculum. The term **integrated** was 
chosen to suggest a course of study representing a synthesis of the student 
and his present environment in which this synthesis extends into the stu- 
dent’s future. 

Part of the rationale for this approach to curriculum development is 
that it should facilitate and assist individuals in synthesizing and 
building concepts, rather than encouraging them only to learn job- 
specific skills. The several state directors and their associ'ites were 
motivated to launch this project as a result of the belief that learning 
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is a continuous process. Learning occurs in every kind of context, of 
course, and it was felt that high school students should look upon life as 
a series of learning experiences. Accordingly, a major goal of the 
project is helping the student to develop productive skills and concepts 
which are useful in many contexts. 

The project will provide a model curriculum, course content, and 
training materials appropriate for use in isolated small high schools 
located in sparsely settled areas where low population density, trans- 
portation factors, lack of financial resources, and other factors do 
not facilitate the development of comprehensive high schools. The pro- 
posed curriculum is intended to provide a general education, systematic 
career selection, and occupational experience in an integrated rela- 
tionship through individualization of instruction. 



Context In Whic h the Project Operates 

Relaticusaips to parent agency . The ICDC Project is one of two 
projects sponsored by WSSSP, the other being Staffing Patterns for Up- 
grading Rural Schools (SPURS). At present, WSSSP is not undertaking any 
projects on its own, as it has in the past several years. It exists 
because of a commitment upon the part of educators in the states of 
Arizona, New Mexico, Colorado, Utah, and Nevada to upgrade rural edu- 
cation and because of governmental and past private support. 

Physical/environmen tal setting . The ICDC Project Director's 
office is located in Salt Lake City with the Utah State Board of 
Education, while the Co-Director has his offices in Carson City with 
the Nevada State Department of Education. As represented by these 
two offices, the entire ICDC Project is widely spread throughout the 
states involved. 

The Director's "of f ice'*‘ is essentially a desk adjoining another 
in an open area of the sixteenth floor of the University Club Building 
in Salt Lake City. This open area is shared by the Utah State Board 
of Education with two or three other agencies. The Director's secre- 
tary shares the same situation as he does, being located around the 
corner within a row of other secretaries. 

The Co-Director's office is more private, being an enclosed area 
in a small office building two blocks from the main street of Carson 
City. He has quicker access to secretarial assistance and the files, 
inasmuch as his secretary and their filing cabinets are outside his 
office in an open area accommodating two or three other secretaries . 

The working conditions in both offices appear to be pleasant. 

The lack of pri/acy in the Salt Lake City office is offset by a 
panoramic view of the northern part of the city, inducing the capitol 
buildings. In contrast, the Carson City office is conveniently located 
near the town's business section, with parking space being available 
just outside the front door of the building. Carson City has the 
drawback of not easily being reached by air transportation. 
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FIG. 1. Project time lines chart 
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Chapter II: Parameters of the Project 



This chapter dicusses the staffing patterns of the project; and 
contains a project organizational structure, a roster of staff, an in-~ 
dex of outputs, and an output map. 

Project Structure 

Staff structure . Essentially a project within a project, ICDC has 
grown out of the activities of WSSSP and in many ways is indistinguishable 
from it. One of these ways is that its objectives are consonant with 
the broader ones of WSSSP, another is in its use of the relationships 
established by WSSSP among people in the five states involved,^ but one 
of the most clear examples is in the organizational structure of ICDC 
as shown in Figure 2, 

Launched in 1962, but not funded as a project since 1970, WSSSP 
acts essentially as an agency, supporting its personnel by SPURS and ICDC. 
These personnel share responsibilities for both projects and WSSSP. The 
Director of SPURS is the Coordinator of ICDC, and the Director of ICDC 
assists him with the SPURS Project, while both of these people operate 
WSSSP. One-half of the Director’s time and one-fourth of the Co-Director’s 
is devoted to ICDC. This means that no professional staff member is 
working full time on the ICDC Project, although one person did last year 
in developing part of the curriculum. 

Project roster . After an initial interview with the Project Director, 
two trips were undertaken for the purpose of interviewing management per- 
sonnel and high school coordinators in the field. A total of six persons 
who have responsibilities for various aspects of the ICDC Project were 
interviewed at sites in Utah, Arizona, Nevada, and Colorado. Those inter- 
viewed were the Project Director, Project Co-Director, Writer-Consultant, 
and three high school coordinators. A project roster of staff is presented 
in Table 1. 



Outputs Generated 

Before the formal interviews were begun, data concerning the project 
were gathered and the outputs being generated were identified. The ways 
in which those outputs are related to each other is shewn in Figure 3. 
Those project personnel who were linked to significant outputs were inter- 
viewed about their roles in generating the outputs. 



^ These involve people from the five states who are interested in 

developing and upgrading rural secondary education. However, as far 
as ICDC is concerned, the strength of this relationship has yet to 
be demonstrated in disseminating and implementing a curriculum. 
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TABLE 1 

Project Roster of Staff by Job Titles 



Administrative Personnel 



Cooperating (Unpaid) Personnel 



^Project Director 
^Project Co-Director 



^Project Ct ordinator at Indian Oasis 
High School at Sells, Arizona 

Project Coordinator at Fredonia 

High School at Fredonia , Arizona 



Project Coordinator at Cloudcroft High 
State WSSSP Director for Utah School at Cloudcroft, New Mexico 



State WSSSP Director of Nevada 

State WSSSP Director for New 
Mexico 

State WSSSP Director for Colorado 
State ICDC Director for Arizona 



Project Coordinator at Weed High School 
at Weed, New Mexico 

Project Coordinator at Panaca High 
School at Panaca, Nevada 

Project Coordinator at Virgin Valley 
High School at Mesquite, Nevada 



Project Coordinator at Tonapah High 
School at Tonapah, Nevada 



Consultants 



^Writer - Consultant 
Evaluation - Consultant 



Project Coordinator at Bryce Valley 
High School at Tropic, Utah 

Project Coordinator at Escalonte High 
School at Escalonte , Utah 



Project Coordinator at Kanab High 
School at Kanab, Utah 



^Project Coordinator at Meeker High 
School at Meeker, Colorado 



Project Coordinator at DeBeque High 
School at DeBeque, Colorado 

Project Coordinator at Clear Creek High 
School at Idaho Springs, Colorado 



^ ' /3 




^Interviewed 
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Index of outputs . T)ie following is an annotated Hst of the 
outputs constituting the principal outcomes of the project. This 
listing includes a classification of each output according to its 
primary focus as cither a research, development, diffusion, or evalua- 
tion effort, as well as an arbitrary identification number cited for 
each output, which identifies it as a management outcome (e.g., M-01) 
or a production outcome (e.g., P-01). 

M-01. Revised Project Management Plan . This is a management 

plan which is periodically reviewed by the Quality 
Assurance Panel. It makes explicit the tasks and time 
lines. Since the inception of the project, the project 
management plan has been modified several times. A 
major decision involved extending the timelines to include 
another year during the second phase (developing the 
curriculum and appropriate instructional materials) . 

This decision was approved by the funding sponsor, 
(development) 

M-02. Fiscal Work Program . A modified Program Planning and 

Budgeting System has been used to handle the budgetary/ 
fiscal problems of the ICDC Project. Essentially, the 
fiscal work program is a set of accounting procedures 
for the project . (development ) 

M-03. Production Procedures . Another part of the project manage- 

ment plan is the set of procedures whereby the curricular 
materials, objectives, and evaluation instruments are devel- 
oped . (d evelopment ) 

M-04. Evaluation Design . A design for evaluating the project's 

testing materials. The field test data is part of the 
management plan. There have been two consultants to the 
project for the evaluation component. (evaluation) 

P-05. Curriculum S tatement . The six months spent at the outset 
of the project's existence in formulating a statement of 
objectives (performance objectives stated in behavioral 
terms) was valuable largely for the experience gained in 
attempting the task. It was decided that the statement be 
repeated, and the directors, writers, and consultants be- 
gan the task again at the beginning of 1969. A curriculum 
statement has not yet been approved by the Quality Assur- 
ance Panel and the funding sponsor. (development) 

P“06. Quality Assurance Panel Recommendations for the Curriculum 

Statement . This is the evaluation — including recommendation 
and decisions — of the curriculum statement (P-05) and its 
operation within the project. (evaluation) 

P-07. Review of Curriculum by Schools . The curriculum statement 
and strategies for implementing an integrated career devel- 
opment curriculum were reviewed by schools during the various 
stages of its development for "field reality checks." 
(evaluation) 

22b . 
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P-08. 



P-09. 



P-10. 



P-11. 



P-12. 



P-13. 



P-14. 



P-15. 



P-16. 
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Instructional Statement . The basis for the curriculum is 
Woodruff’s Life Internship Model. A theory of instruction 
springing from this model has been developed during the 
life of the project. (development) 

Quality Assurance Panel Recommendations for the Instructional 
Statement . This is the evaluation — including recommendations 
and decisions — of the instructual statement (P-08) and its 
operation within the project. (evaluation) 

Specifications for Single Concept Learning Unit Prototypes . 
Prototypes of the single concept learning units were deve- 
loped by a process whereby specifications for the units 
were drawn up, the specifications for the prototypes were 
reviewed by the Quality Assurance Panel and the evaluation 
consultant, the production of the single concept learning 
units was begun on a priority basis, revisions of the units 
were made and reviewed, the prototypes were tried out in 
schools, and subsequently revised. (development) 

Quality Assurance Panel Recommendations for SCLU Proto - 
types . This is an evaluation — including recommendations and 
decisions — of the SCLU prototypes (P-10) and their opera- 
tion within the project. (evaluation) 

Analysis of Baseline Data from Schools . Information about 
the high schools’ students was acquired prior to the comple- 
tion of the SCLU prototypes. It was believed that data 
concerning the students’ interests, backgrounds, and abilities 
were needed if the prototypes were to "fit” the students 
in the rural high schools. (evaluation) 

Single Concept Learning Units . The initial thrust in pro- 
ducing SCLU prototypes was in the area of basic technology. 

The other two areas of society and work and career guidance 
have received more emphasis recently. The prototypes for 
all three areas are expected to be completed by the end of 
the summer of 1971. (development) 

Field Trials in Schools . There actually will be two types 
of field testing of the prototypes. Currently, a sel*. 
group of SCLU prototypes are being tested in some of the 
high schools as part of the plan to see if they are appro- 
priate and effective; these do not constitute the formal 
field trials. The formal testing will take place when 
all of the SCLU prototypes have been written. (evaluation) 

Analysis of Field Test Data . When the field trials have 
been completed, the data will be analyzed by the evaluation 
consultant and the project directors. (evaluation) 

Specifications for Revised Single Concept Learning Unit 
Prototypes * In the development of the SCLU prototypes, 
a critical stage occurs after the field trials have been 




670 



finished and the results of this testing are reviewed by 
the evaluation consultant, the directors, and the Quality 
Assurance Panel. Revisions of the units will be made in 
accordance with the critiques and recommendations of 
these persons in order that the prototypes be as useful 
and effective as possible for rural high school students, 
(development) 

P- 17 . Quality Assurance Panel Recommendations for Revised SCLU 
Prototypes . This is an evaluation — including recomenda- 
tions and decisions — of the revised SCLU prototypes (P-16) 
and their operation within the project. (evaluation) 

P-x8. Curriculum; Single Concept Learning Units Package . The 

principal focal output of the project is the package of 
single concept learning units which will be the result of 
the writing, testing, rewriting, retesting, and rewriting 
which takes place during the life of the project. (devel- 
opment) 

Table 2 is an indexing of all the identified outputs of the ICDC 
Project by level, i.e., whether the output is focal, component, or facili- 
tating in its nature. A focal output is an outcome of work expected, by 
contractual obligation, to emerge from a project. A component level out- 
put is an outcome of work effort that constitutes an element of or one 
step in the approximation to a focal output, and a facilitating output is 
an outcome of work effort that is supportive to the development of any 
of the outputs listed above, but is not in itself an instance of such 
outputs. 

Output map . Figure 3 is an output map showing the dependent rela- 
tionships between the outputs of the project. The relationships identi- 
fied are not necessarily sequenced over time. 
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TABLE 2 



Output Index 



LEVEL I: FOCAL OUTPUTS 

P-18. Curriculum: Single Concept Learning Units Package. 



LEVEL II: COMPONENT OUTPUTS 

P-13. Single Concept Learning Units 

P-16. Specification for Revised Single Concept Learning Unit Proto- 
types. 

P-10. Specifications for Single Concept Learning Unit Prototypes. 
P-08. Instructional Statement. 

P-05. Curriculum Statement. 



LEVEL III; FACILITATING OUTPUTS 

P-17. Quality Assurance Panel Recommendations for Revised SCLU Pro- 
totypes. 

P-15. Analysis of Field Test Data. 

P-14. Field Trials in Schools. 

P-12. Analysis of Baseline Data from Schools. 

P-11. Quality Assurance Panel Recommendations for SCLU Prototypes. 
P-09. Quality Assurance Panel Recommendations for Instructional 
Statement . 

P-07. Review of Curriculum by Schools. 

P-06. Quality Assurance Panel Recommendations for Curriculum State- 
ment . 

M-03. Production Procedures. 

M-02. Fiscal VJork Program. 

M-01. Revised Project Management Plan. 

M-04. Evaluation Design. 
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Chapter III: Summary of Data 



The tables included in this chapter each summarize the interview 
data by category and by output interviewed around during the site 
visit. The statements made during an interview around a particular 
project output are identified and then classified under the cate- 
gory headings of (^a) the standards by which one judges the satisfac- 
tory completion of the output, (^) the tasks required to generate an 
output meeting those standards, or (£) the enablers (knowledges, skills, 
and sensitivities) which facilitiate the carrying out of t^ose tasks. 

Within each category are a series or set of descriptive labels 
which are representative of interviewee statements within a particular 
category. These descriptive labels are listed in the table under the 
category heading. In the process of reducing raw data, narrative 
interviewee statements about an output were linked to one of the three 
categories (i.e., standard, task, or enabler). Each narrative state- 
ment was then classified by means of a number code according to the 
most representative descriptive label within a given category or sub- 
category , ^ 

The production standards cited by the interviewees are shown in 
Table 3. Interestingly enough, the same standard was not applied to 
more than one product. 

Table 4 shows how often management standards were cited by the 
interviewees in discussing their tasks. The data reflect the impor- 
tance of project-school relationships. 

Table 5 implies that the knowledges required of the project staff 
came largely by education courses and situational contexts. 

The skills necessary to '’^-omplish the outputs of the project are 
dispersed rather widely throi^hout the series of descriptive labels 
within the skills category. This can be seen in Table 6. 

Table 7 shows that only three sensitivities were cited by the 
interviewees. The number of tasks per output are shown in Table 8. 



^ If the reader is interested in the narrative statements of the 
interviewees (raw data), these can be found in the Appendix. To locate 
the narrative statement for any given category, first note the out- 
pu t and its identification number in the table. Second , note that each 
descriptive label within a given category has a distinct number or code. 
Turn to the Appendix and locate the output. Under the output locate the 
category label or heading (standard, task, or enabler) and pinpoint the 
number or numbers (depending on frequency cited) of the descriptive 
label which appeared in the table. The statement in the Appendix op- 
posite this number is the original narrative statement from an inter- 
viewee and is only represented in the tablc^jb^ the descriptive label 
and its number coding. 
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Chapter IV: Supplementary Data 



Included in this chapter is information about the classifications 
of output characteristics, the backgrounds of project and agency per- 
sonnel, the job requirements for the project, the support resources of 
the project, and its funding. 



Classifications of Output Characteristics 

As the Oregon Studies evolved it became evident that outputs could 
be categorized in terms of a number of variables. Among them are 
(^) Structure (product, event, or condition), (^) Function (policy setting, 
management, or production), {c) Level (focal, component, or facilitating), 
(^) Character (knowledge, technology, implementation, or information) , and 
(^) Stage of completion. These five schema are represented in Table 9 for 
each project output identified, with frequencies summarized for each 
category. Table 9 has been added to this profile subsequent to the 
profile's original writing. 



Summary of Staff Backgrounds 

The principle staff members of the project are also its administra- 
tors. As is the case with a number of other similar projects, much of the 
production work is "hired out." Six persons were interviewed, however, 
and their various backgrounds provide an index regarding the kinds of 
individuals who carry forth projects such as ICDC. 

The highest degree held by all but one of the interviewees was a 
master's degree. One person holds a doctorate, and he has been used as a 
curriculum-consultant coordinator. The others might be characterized as 
professional administrators. Because of the structure of the staff, it is 
not really worthwhile to compare salaries with academic degrees. The 
Director and Co-Director receive the same salary rate, and all other inter- 
viewees are presently employed by other agencies. 

By far the most common work experience of the six interviewees was 
teaching and/or administering in the public schools (see Table 10). 
of those interviewed have had five or more years in the public schoL'i, 
and three of these individually have had from 18 to 21 years experience. 

It would seem that none of the interviewees has received specific formal 
training for work in an educational development or diffusion project. 
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TABLE 9 

Claaaificatlona of Oucpuc Characcarlatica 



O utput Characcarlatlc* 





Prolact Outputa 


Structure 


function 


Level 


Character 
(Products onlv) 


Completion Stage 


No. 


Label 


P 


e c 


ps n 


p 


f 1 c {7 


k !. ll l3 


12 3 4 5 6 


M-01 


Reviacd Project Management. Structure 




X 


X 




X 




X 


M-02 


Fiacal Work Program 


X 




X 




X 


X 


X 


M-03 


Production Proccdurca 


X 




X 




X 


X 


X 


M-04 


Evaluation Dcaign 


X 




X 




X 


X 


X 


P-05 


Curriculum Statement 


X 






X 


X 


X 


X 


P*06 


Quality Aaaurancc Panel Recoanenda- 
tlona for Curriculum Statement 


X. 




X 




x‘ 


X 


X 


P-07 


Review of Curriculum by Schoola 




X 


X 




X 




X 


P-08 


Instructlonel Statement 


X 






X 


X 


X 


X 


P-09 


Quality As*:.urancc Panel Recommenda- 
for Inatruptlonel Starement 


X 




X 




X 


X 


X 


P-10 


Specif Icatlona for Single Concept 
Learning Unit Prototypes 


X 






X 


X 


X 


X 


P-11 


Quality Aaaurancc Panel Rccomaendatlona 
for Single Concept Learning Unit 
Prototypaa 


X 




X 




X 


X 


X 


P-12 


Analysis of Basallnc Data from Schoola 


X 




X 




X 


X 


X 


P-13 


Single Concept Learning Unlta 


X 






X 


X 


X 


X 




Field Trlala In Schoola 




X 


X 




X 




X 


P-15 


Analyala of Field Teat Data 


X 






X 


X 


X 


X 


P-16 


Specifications for Revised Single Con- 
cept Learning Unit Prototypes 


X 






X 


X 


X 


X 


P-17 


Quality Assurance Panel RccoaMendatlons 
for Revised S.C.L.U. Prototypea 


X 




X 




X 


X 


X 


P-18 


Curriculum Single Concept Learning Unlta 
Packasc 


X 






X 


X 


X 


X 


Claaalflcatlon Frcoucnclca^ 


15 


2 1 


0 11 


7 


1 3 14 


0 9 0 6 


1 10 1 2 4 0 



* Th« apaclflc output characcarlatica art Idaotlflad aa follova: 



Structure Functloo 



Laval 



Character 



Cowplatloo Stage 



b 



p - product 
a - event 
c - condition 



pa “ policy acttlng 
■ - maiugeacnt 
P - production 



fl - focal 
c - eoaponent 
t2 ” fecllltacing 



k * knotfladgc i 

t “ technology 2 

1| - lapleaentatloa 3 

I 2 “ Inforwatlon 4 

S 



6 



Diti totili In thli tibl* uy viry illghtly from diCi In cnblcn rcporctd clncuhcr*. Thin 
claiilElcitlon of ouepuen hiving been reviled and applied to Chcic data lubicqut 



• coaplctcd ovar one year ago 

- conplctcd 3 to 12 montha ago 

- conplctcd within laat 3 aoa. 

• currently in progrcaa 

- not yet underway 

- on going (contli^uoua) 

a function of declalon rulea 
CO the preparation of the 
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TABLE 10 

Selected Sumn\ary of Interviewee Backgrounds 



Years of Prior Experience 


Number 


of Interviewees in Each Year Category 




None 


Less than 1 


1-4 Years 


5 or More 
Years 


In College or University 










Teaching 


4 


0 


2 


0 


In College or University 










Research 


6 


0 


0 


0 


In Public Schools 

In State or National Education 


0 


0 


1 


5 


Agencies 


3 


0 


0 


3 


In R & D Centers 


6 


0 


0 


0 


In Present Organization 

(may be concurrent with other 










areas above) 


2a 


0 


3 


1 



^Indicates that two people did not respond to this category. 



In general, the interviewees reported that their professional training 
had been of value to them in carrying out their responsibilities in the 
project and that their professional work experiences were important in pre- 
paring them for their project jobs. Academic courses such as psychology, 
curriculum development, learning theory, testing, and research methods were 
mentioned as being helpful. It is Interesting to note that many of these 
same courses are considered by the interviewees to be requirements for their 
positions on the project. Thus, it would appear that (£) they feel ade- 
quately qualified for their jobs, and (^) academic courses do prepare 
individuals for engaging in sets of activities encountered on projects such 
as ICDC. Only the Project Director indicated that his position might re- 
quire greater academic preparation than he had acquired. The others said 
that the academic degree necessary for their job was the one they held. 
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Chapter V: Project Dynamics 



The two key administrators of the ICDC Project have been associated 
with the parent agency project, WSSSP, for several years, the Co-Director 
having been appointed Coordinator of WSSSP about two years ago. Their 
working relationship basically involves supporting each other in the two 
current projects which WSSSP sponsors, namely, ICDC and SPURS. Although 
one person is in Salt Lake City and the other is in Carson City, they 
seem to experience little difficulty in communicating with one another 
and in arranging meetings for purposes of decision making. 



Interrelationships Among Project Personnel 

Aside from the established relationships of the Director and Co- 
Director, there are three sets of relationships among ICDC personnel. 

At a political level, the Director deals with the five state superin- 
tendents and the 14 high schools. The second set of relationships in- 
volves the Director with those persons who are consultants to the project, 
that is, the members of the Quality Assurance Panel and certain other 
consultants such as evaluation consultants who give direction to the 
project goals and modus operand! . 

The third set of relationships involves the Director with curriculum 
writers. Last year (1969-1970) a head writer coordinated the writing 
of the general technology units while he was on a leave of absence from 
his position as Specialist in Vocational Education and Special Education, 
but he has returned to his job and is now acting as an occasional writer 
for the project. The Director is coordinating the writing assignments 
at present, and he is especially overseeing the career guidance units. 



Interrelationships with Sponsors 

A significant factor in the birth and growth of the ICDC Project has 
been the continuous support which WSSSP, the parent agency project, has 
received from its sponsors over the years. WSSSP’s principle sponsor was 
generous in its support over a relatively long period of time, enablin<^ 
the network of state relationships upon which ICDC depends to become 
firmly established. The sponsor of ICDC is a different agency from tne 
principal sponsor of WSSSP, but it would appear that there will be no 
loss of continuity as a result of the departure of the old sponsor from 
the WSSSP scene. 



Interrelationships With Other Agencies 

In as much as ICDC exists by virtue of its relationships with a 
number of differing agencies, it is necessary to indicate what the structure 
of those relationships is like in order to understand the project. The 
Director and Co-Director are legal employees of the Utah State Board of 
Education and the Nevada State Department of Education respectively, being 
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assigned to WSSSP and ICDC. Other baeic relationships are those that 
exist between ICDC and the departments of education of the five states. 

The Utah State Board of Education and the Nevada State Department of 
Education provide facilities for the Director, Co-Director, and supporting 
staff. Another set of relationships upon which the ICDC Project is built 
is with the 14 cooperating high schools in the five states. At this point 
in time, the persons who are coordinating the program are mainly the key 
administrators of the 14 school districts. A Career Selection Agent will be 
appointed by the administrator of each high school to actually implement 
the program. The State Project Directors receive one-fifth of their 
salaries from the ICDC Project; and, aside from the Director and Co-Direc- 
tor, they are the only other professionals on the payroll. 

Staff Background 

ICDC’s Director and Co-Director have had public school careers prior 
to becoming involved in WSSSP. One had been a public school teacher and 
administrator for 18 years and the other had been a mathematics teacher 
for five years before becoming a mathematics consultant for his state 
department of education. Their skills appear to be complementary, and 
both have experience in project management. 

Management Structure 

The lines of authority and responsibility are fairly well defined 
for the ICDC Project. The Director has responsibility for developing 
the curriculum and coordinating the time lines of the project. The 
Co-Director is the person who attends to the fiscal matters of the pro- 
ject and who helps formulate policy. In as much as the two professionals 
get along well together and are able to communicate adequately at long 
range (that is, from Salt Lake City to Carson City), the structure is 
simple and effective. The morale of the staff is high partly because of 
the congeniality of the individuals and partly because they believe in 
what they are doing, namely, attempting to improve rural education in 
the five states. 
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Chapter VI: Implications for Training 



Most of the suggestions for training offered by the interviewees 
were general in nature. Since most of the interviewees held administrative 
positions, the experiences they felt would be valuable for persons intending 
to work on projects similar to ICDC are largely those needed in administra- 
tive or managerial work. Here are their summarized suggestions for training 
individuals to assume their kinds of responsibilities: 

1. Training in evaluation techniques. 

2. Training in curriculum development. 

3. Experiences in both elementary and secondary schools. 

4. Experiences in counseling. 

These were the major knowledges the interviewees felt would be important 
for persons engaged in projects such as ICDC: 

1. An understanding of educational objectives. 

2. A familiarity with management techniques such as PERT (Program 
Evaluation and Review Technique) and PPBS (Program Planning 
and Budgeting System) . 

3. An understanding of children. 

The interviewees mentioned these specified skills as being valuable 
for someone assuming their jobs: 

1 . Political skills . 

2. The ability to get along with people. 

3. Communications skills, including writing skills. 

4. An ability to work with tests and student records. 

The interviewees also recognized certain sensitivities or personality 
traits as being important to individuals engaged in educational development. 
These included: 

1. Being creative and innovative. 

2. Being forward looking. 

3. Being able to tolerate differences of opinion. 

4. Having a conviction that education can be Improved. 

5. Being dedicated to helping people. 

6. Having a "large" view of the world. 

Experience on the Job itself, or experience performing tasks which 
are similar or identical to those of the project, was felt to be the 
most valuable kind of training an individual could receive# 
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Appendix: Listing of Output Standards, Tasks, and Enablers 



The following is a list of standards, tasks, and enablers for outputs 
around which interviews were conducted. These statements were extracted 
from discussions with interviewees and were coded into their respective 
category sets. The selected code precedes the statement and Indicates the 
following for: 

STANDARDS 

Code Jr Structure of Standards. 

J-1 Standards against which outputs are judged. (output 

oriented) 

J-^ Standards against which processes and/or operations are 

judged. (process oriented) 

Code LM: Primary Categories of Standards. 



TASKS 

Code NO: Clusters of Taskst 



ENABLERS 
Code S: 
S-1 
S-2 
S-3 

Codr' UV: 



Structure of Enablers. 

Knowledge. 

Skill or ability to perform. 

Sensitivity or awareness. 

Primary Categories of Enablers (knowledges, skills, or 
sensitivities) . 



The codes associated with these three categories (standards, tasks, 
enablers) are the same both here in the listing and as previously cited in 
Chapter III tables. 

The interviews of six persons involved in the ICDC Project were 
conducted over a period of two months ut five different sites. With 
some notable exceptions, the interviewees were concerned with three dif- 
ferent kinds of responsibilities: management, curriculum development, and 

coordination of the project’s activities in the schools. 

The following is the cataloguing of the standards, tasks, and enablers 
associated with the outputs which served as foci for the several inter- 
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views. Each output is identified by its arbitrary identification number, 
title, level, and status. By level is meant whether it is a focal com- 
ponent, or facilitating output, and by status whether it was not yet 
initiated, in process, or completed at the time of Interview. 



M-01: Revised Project Management Plan 

Level: Facilitating 

Status: in process 



One of the major responsibilities — perhaps the major responsibility — 
of the project management has been to find people who can actually write 
the Single Concept Learning Units. Coordinating the conceptualization, 
writing, and testing of the curriculum has proved to be tremendously 
difficult task, especially since the budget for the project is a fraction 
of the aa»v>’:nt which would be spent by a commercial publisher. 

In this project, the Co-Director's role is a supportive one. He 
reviews the conceptual papers written by the Director and the project 
consultants. There is a further review by the Qaility Assurance Panel. 
Apparently there is an excellent network of communications among these 
persons, in spite of the fact that they live in states separated by 
hundreds of miles. 

STANDARDS: 

J LM 

1 13 Approval by the Quality Assurance Panel for major statements, 

procedures , etc. 

2 C4 Schedule is met to match particular milestones of a product. 



TASKS : 
NO 



02 

02 

33 

24 



24 

21 



24 



24 

24 

21 



Confer with Director about plans for the project. 

"Redesign the project" (with the Director). 

Decide "who is going to get what job" and how he will be paid. 
Critique papers written by the Director and members of the 
panel. 

Review and evaluate work which is done by contract. 

Identify people in the field who can write appropriate 
curriculum materials. 

Direct (supexrvise) individuals who can write desired curri- 
culum materials. 

Set up a review procedure for units that writers produce. 

Set up guidelines for writing curriculum unit. 

Identify people who can tier objectives — taking overall 
objectives down through a series of tiers to instructional 
level objectives. 
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24 Direct people (supervise) who can tier objectives in pre- 
paring units of instruction (set up specifications for 
curriculum) . 

22 Coordinate activities of writers, consultants, and people in 
the high schools. 



ENABLERS : 

S UV 

2 02 Be able to deal with people. 

2 18 Relating all aspects of project to a basic philosophy of 

education. 

2 24 Be able lo interpret project's goals in terms of Woodruff's 

Life Internships Model at the instructional level. 

2 03 Beableto write examples of curricular materials . 

2 03 Be able to write curricular objectives. 

2 37 Be able to identify people who can tier objectives — taking 

overall objectives down through a series of tiers to instruc- 
tional level objectives. 

2 29 Be able to direct people who can tier objectives. 

2 37 Be able to identify people who can write units of instruction. 

1 05 Understanding of the Woodruff Life Internship Model. 

2 34 Skill in coordinating activities of writers and consultants. 

2 05 Skill in using PERT technique . 



M-02: Fiscal Work Program 

Level: Facilitating 

Status: In Process 



One of the advantages of a project which is run by two persons is 
that a high degree of efficiency can be achieved when there are defi’^^' ^ 
lines of authority and responsibility established. The fiscal matterL 
of the project are entirely the responsibility of the Co-Director, and 
his fiscal work program apparently is operating without difficulty. 



STANDARDS : 

J LM 

1 14 "If reports are accepted by sponsor," the fiscal job is done 

well. 

1 09 If .accounts are in order and balanced, the accounting is 

satisfactory. 

2 13 Work program printout shows "we are staying within line 

categories ." 



TASKS : 



NO 



22 Keep financial records for the project. 

22 Have m«jjo accounts posted (because federal categories and 



state accounting system don’t match). 

22 Keep up with sponsor's accounting procedures. 
22 Report accounts of project to sponsor. 



22 Confer with Director about what will be spent (in the individual 
categories) . 

22 Analyi:e what has been spent (and what is remaining) for the 
State. 



2 27 Abil'^ty to use a calculator. 

2 27 Skill in using PPBS techniques. 



P-13: Single Concept Learning Units 

Level: Component 

Status: In Process 



The person who has been responsible for the production of the basic 
technology area of the curriculum has had tasks which are similar to the 
Director's. The latter has coordinated the other areas of the curri- 
culum, in addition to his management duties. 



ST/JNDARDS : 



1 12 Writer's unit follows format for basic technology units. 



22 Have secretary post expenses. 

22 Have secretary send out checks after expenditures have been 



made. 

22 Work with secretaries setting up ledger sheets and categories. 



ENABLERS : 



S UV 



1 01 Knowledge in math and math instruction. 



J LM 



1 28 Students can follow the unit and perform satisfactorily. 

1 11 Sequence of tasks for learner is appropriate. 




TASKS: 



NO 

21 Identify people with expertise in each of eight areas (general 
mechanical, spatial , electronics , etc. ) • 

03 Set up a design for the single incremental learning units for 
each area of the capability. 

21 Train the writers in the construction of the design. 

06 Redesign (change) the units after writers submit them so that 
the units are workable for students. 

05 Test certain units in the local schools to see how effective 
they are with students of low ability and/or motivation. 

05 Test skills of high school students. 

06 Determine if students were adequate to perform basic skills — 
if they were, units for their skills were not written (such 
as basic computation). 

05 Field test units in three schools. 

21 Identify people who can write curriculum materials and want 
to do so. 

24 Send out units for review by experts in vocational areas. 

06 Redesign units that proved unsatisfactory in initial field 
trials . 



ENABLERS : 

S UV 

1 03 Knowledge of various types of curriculum development. 

2 01 Skill in teaching at the high school and higher levels. 

2 01 Skill in training people. 

1 02 Understanding of how children learn. 

2 06 Ability to organize sequential patterns in accordance with 

the way students naturally learn. 

2 03 Ability to design behavioral objectives. 

2 39 Ability to translate behavioral objectives into verbal test 

items . 

2 39 Ability to pull elements out of curriculum and put them 

into a pretest and a oosttest. 

1 05 Knowledge from development of learning programs (courses) to 

solve problems in industry. 



P-14: 


Field Trials In Schools 


Level : 


Component 


Status: 


In process ' 



248 



696 



The Single Concept Learning Units which have been field tested are 
those, like the one dealing with drafting, that have been developed in 
the general technology area of the Integrated Career Development Curri- 
culum, That part of the curriculum was the furthest toward completion at 
the time of interview. The task of field testing has proved to be more 
difficult for some school administrators than for others. As it hap- 
pened, two of the three persons who were charged with the responsibility 
of seeing that the trials were conducted had trouble securing the coopera- 
tion of teachers who would devote the time and energy required to test 
the units adequately. It was reported that other administrators experi- 
enced little or no difficulty in getting the units tested. 



STANDARDS: 

J LM 

1 12 Writer’s unit follows format for basic technology units. 

1 28 Students can follow the unit and perform satisfactorily. 

1 11 Sequence of tasks for learner is appropriate. 

1 22 The packages work well for the individual student. 

1 24 The package is understandable to the student. 

1 22 The package does what it is supposed to do— the student is 

able to perform in the ways prescribed. 

2 16 Good communications between project personnel and high 

schools. 

2 17 Enthusiasm on the part of students for the learning kit. 

2 16 Approval by a student advisory committee. 

2 16 Approval by a teacher advisory committee. 



TASKS: 

NO 

21 Identify people with expertise in each of eight areas 
(general mechanical, spatial, electronics, etc.) 

03 Set up a design for the single incremental learning units 
for each area of capability. 

21 Train the writers in the construction of the design. 

06 Redesign (change) the units after writers submit them so. 
that the units are workable for students. 

05 Test certain units in the local schools to see how effective 
they are with students of low ability and/or motivation. 

05 Test skills of high school students. 

06 Determine if students were adequate to perform basic skills; 
if they were, units for these skills were not written (such 
as basic computation). 

05 Field test units in three schools. 

21 Identify people who can write curriculum materials and who 
want to do so. 

24 Send out units for review by experts in vocational areas. 

06 Redesign units that proved unsatisfactory in initial field 
trials. 
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21 Expose career selection agent, guidance person, and high 
school principal to ICDC package. 

06 Convert individual units of package to semester-hour values. 

05 Each school principal assigns individual units of packages to 
students as projects. 

22 Each school principal has teachers check to see how students are 
doing with their packages. 

01 Guidance person determines what students can do (e.g., the 
case of a brain-injured student). 

05 Recommend to the teacher the units to use with her students. 

05 Recommend the kinds of materials which will help students to 
supplement their on-the-job experiences. 

05 Participating teachers insert packages in the curriculum to 
replace other units — for whole class or individuals. 

21 Career Selection Agent, principal, and guidance person expose, 
through workshops or person-to-person contacts, the entire 
faculty to ideas and materials provided at workshop on imple- 
mentation of packages. 

05 Help each student to explore his vocational interests by ad- 
ministering appropriate tests . 

29 Help each student to assess his abilities. 

03 Select students to try out preliminary kit (drafting). 

22 Assign teacher to assist students in working with kits. 

31 Serve as a consultant to the persons directing the project. 

29 Encourage principal and teacher of a high school to use the 

drafting kit. 

29 See that the curriculum materials are available at the high 
school. 

21 See that the teachers of the high school are able to avail 
themselves of the ICDC training program by making time 
and money available to them. 

24 See to it that there is a follow-up to the training program 
and that there is feedback from the teachers. 



ENABLERS: 

S UV 

1 02 "Understanding of educational objectives as propounded by 

Mager." 

3 02 Appreciation for the problems of the small, isolated commun- 

ities . 

3 02 Realization that for the most part one is not able to give 

first-hand experiences to students in the many vocational 
fields. 

1 02 Knowledge from education courses, especially counseling. 

1 04 Experience as a counselor, getting to know child and his 

family. 

1 06 Knowledge of students outside the classroom. 

2 02 Ability to work with faculty. 

1 06 Wide variety of experiences in all aspects of a small rural 

school. 
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2 01 Teaching experience — from fifth grade through graduate school, 

1 06 Exposure to innovation and change as a result of involvement 

with WSSSP. 

2 30 Being adaptable to change and experimentation, 

3 lb Awareness that schools should be doing a better job (**are 

not doing the best job by any means**). 
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CASE PROFILE NO. 13 



Written by 
R. E. Myers 



PROJECT TITLE: The Development of Protocol Materials 

(PROTOCOL Project) 



AN EDUCATIONAL DEVELOPMENT PROJECT CONCERNED WITH: Develc .Lng 

visual, auditory, or printed reproductions of human behaviors 
that portray concepts found in the professional practice of 
teaching and learning. 



A PROJECT OF: Teaching Research Division 

Oregon State System of Higher Education 
Monmouth, Oregon 97361 
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This profile has been prepared according to 
PROFILE FORMAT No. 2 

Three profile formats are represented in this volume. 
The reader should refer to this number when making 
use of the reader's GUIDE to the profiles. 
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Chapter I: Overview 



This chapter contains a synopsis of the protocol materials project; 
the rationale, objectives, and significance of the project; and a de- 
scription of the context in which the project operates. 



Synopsis of the Project 

Title: The Development of Protocol Materials. 

Responsible Institution: Teaching Research Division of the Oregon 

State System of Higher Education. 

Funding Source: U.S. Office of Education, Bureau of Educational 

Personnel Development. 

Funding Duration: July 1, 1970 to June 30, 1971. (12 months) 

Observation Date: October 1970. (reobserved in June 1971) 



Present Stage of Development: Initial phase. 



RDD&E Focus of Project: Educational development. 



Expected Outcomes: 1. 

2 . 

3. 

4. 

5. 

6 . 

7. 

8 . 



Articles for Teaching Research Newsletter 
Final Report 
Progress Report 
New Proposal 

Prospectus for Refunding 
Protocol Films 
Field Test Evaluation Guide 
User Materials 



Level of Funding and Duration: Low-Medium, (level 2 of 7 levels) 



Agency Setting: College/University. 



Staff Summary (Current) : 

Total Full Time Equivalency (in 
man years) : 

Number of Personnel Assigned: 
Prime Contractor 
Others (Film Production) 
Consultants 



Professional 



3.25 

7 

2 

3 



Support 

1.25 

3 



Professional Specialties of Staff (interviewees only) : educational 

psychology, instructional media, research design, anthropology, 

English education, communications, educational measurement, curriculum 
and instruction, sociology, social studies education, and electronics. 
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Slgnlf Icancet Ratlonale> and Objectives of the Project 

The Development of Protocol Materials Project (Protocol) found signifi- 
cance in the idea for protocol materials advanced in Teachers for the Real 
World (Smith, B.O. et al., 1969). As defined in that book, protocol materials 
are visual, auditory, or printed reproductions of human behaviors that por- 
tray concepts found in the professional practice of teaching and learning. 
Therefore, virtually all types of professional situations in which a teacher 
is likely to be involved are considered appropriate for reproduction through 
protocol materials. As instructional materials they are intended to 
facilitate teachers in developing: 

1. A functional knowledge of those key concepts generally identified 
in teacher education as relevant to the work of a teacher (e.g., 
commitment to learning, constructive sense of self, analysis, and 
evaluation) ; 

2. The ability to discover, interpret, and describe these concepts 
as they are found in the behavioral situations arising within the 
teacher's world of work; 

3. The ability to use such understanding in formulating plans that, 
as alternatives to current plans, are more effective in reaching 
desired ends in teaching; e.g., conferences with parents or other 
interactions between the teacher and students, teachers, parents, 
or administrators. 

The rationale of the Protocol Project is to provide a means whereby 
a teacher can not only learn to clarify the desired ends toward which he is 
working within a specific situation, but can also learn to develop strategies 
for reaching those goals. Protocol materials should enable the teacher to 
focus on specific behaviors relevant to the concepts found in a situation, 
and from such a study acquire the cognitions and affects necessary for him 
to make the decisions appropriate to bridging the gap between the goals 
identified and the strategies needed to reach them within a particular 
situation. Although protocols can take various forms, they must be able to 
communicate among trainees in teacher education. Film, video tape, film 
strips, and audio tape are examples of media which may be used to represent 
or provide concrete referents for the concepts being studied. 

Teaching Research (TR) is one of 12 institutions to receive a grant to 
develop protocol materials. The TR project has the following distinguishing 
characteristics : 

1. It is designed to obtain information about the methodology of 
protocol materials development. 

2. It is to produce an exemplary design for evaluating the effective- 
ness of the protocol materials developed in ongoing teacher 
education programs. 

3. It is to produce an exemplary design for field testing the protocol 
materials for their utility in a wide range of institutions having 
teacher education programs. 
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4. It is to demonstrate a unique linkage between institutions and 

agencies within a state and across the nation for the development 
and testing of the materials to be produced. 

The objectives of the project are to: (a) develop a functional and 

relatively exhaustive approximation to conceptual frameworks for classify- 
ing learner outcomes and to use the frameworks in forming a set of high 
priority educational outcomes; (_b) develop protocol materials illustrative 
of the class of priority outcomes identified in [a]; (c) prepare user’s 
guides for protocol materials developed in [b]; and (d) prepare a report 
summarizing the feasibility of developing and using protocol materials 
under widely varying conditions. 

The conceptual frameworks, protocol materials, and guides are intended 
to undergo field trials in the contexts for which they were specifically 
designed, and (b) contexts for which they were not specifically designed. 

The purpose of these trials will be to assure that the protocol materials and 
accompanying guides will be as representative and generalizable as possible. 
In an effort to achieve these qualities the frameworks, protocol materials, 
and user’s guides will be developed in the contexts of two quite different 
kinds of institutions; a college of education and a clinical high school. 

The project focuses primarily on the methodology of protocol develop- 
ment and the provision of validated exemplar protocol materials, rather than 
on the production of large amounts of material. Also, it focuses on protocol 
materials representing validated classes of pupil outcomes rather than on 
classes of teacher behaviors assumed to be related to pupil outcomes. 

Context in Which the Project Operates 

R elationships to parent agency . The Protocol Project is administered 
by the Learning Ecologies Program of Teaching Research. This program has 
been especially interested in the field of teacher education. Teaching 
Research, a division of the Oregon State System of Higher Education, renders 
a variety of services to the state and also to the nation in the fields of 
educational research , development , and evaluation . 

Relationships to sponsor . The funding agency for the project is the 
U.S. Office of Education (USOE). The project has worked closely wi^b jSOE’s 
Bureau of Educational Personnel Development (BEPD) and its agency, the 
Leadership Training Institute (LTI). In addition to regular LTI staff 
members at the University of South Florida in Tampa, the LTI is comprised of 
14 individuals concerned with innovation in teacher education; these persons 
reside in all geographical areas of the country and operate individually 
and collectively to provide leadership for a variety of projects, such as 
Protocol, sponsored by BEPD. 

Physical/environmental settings . The Protocol Project at Teaching 
Research is being administered through the cooperative efforts of three 
institutions; Teaching Research, Oregon College of Education (OCE) , 
and John Adams High School. Both TR and OCE are located in Monmouth, Oregon. 
Adams High School is located in Portland, Oregon, some 65 miles north of 
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Monmouth. Adams is in a soclo-economicaliy mixed neighborhood in t\\e 
northestern part of Portland, and is in Its second year of operation as 
an avowedly experimental high school in an urban setting. Teaching 
Research has office space In the Education Building on the OCE campus, 
as we^l as in five other campus buildings nearby. It is within this 
physical setting that the project is housed. 

Relationships to other efforts of a coordinated program . Teaching 
Research’s Protocol Project is one of 13 such projects that have been funded 
by the BEPD. Meetings for the project directors and LTI staff have been 
scheduled for the fall of 1970 and for the spring of 1971. It is antici- 
pated that communications among the various projects will be frequent and 
productive. Although the LTI staff is providing conceptual and technolo- 
gical assistance for these initial efforts to produce protocal materials, 
each project is essentially autonomous. 

The contextual map (Figure 1) graphically represents the preceding 
relationships between the agencies involved in the Protocol Project. 
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Chapter II: Parameters of the Project 



This chapter contains the staffing pattern of the Protocol Project, 
descriptions of the products being produced, and an index of management 
responsibilities . 



Project Structure 

Staff structure * The organizational map of the project (Figure 2) 
indicates the major relationships among the personnel of the institutions 
involved in conducting the project. The basic relationships which were 
established at the outset of the project have been maintained thus far. 

Project roster. Five project members were exteisively interviewed. 
These interviewees included the Project Director, the Co-Director, the 
OCE Coordinatjr, the Adams Coordinator, and one of the two research 
assistants (see Table 1) . 

The support personnel for OCE and Teaching Research staff members 
consist of a secretary (half-time) at each institution and a media tech- 
nician from Teaching Research. 

Film making is largely the concern of the Co-Director, whereas the 
overall management of the project and design of the field testing procedures 
is being handled by the Project Director. The Director of the Learning 
Ecologies Program (Special Resource Person, Figure 2) is giving leadership 
to efforts to formulate a conceptual framework for the project, especially 
with regard to learner outcomes. 

Outputs Generated 

The outputs identified in this study are sorted into two categories: 
production oriented and management oriented. A production-oriented outcome 
is defined as a tangible or "hard" result of work efforts, surviving in the 
form of a transportable product. A management-oriented outcome is defined 
as an identifiable result of work effort, directly related to management 
operations . 

Through contact vith various project personnel, the tangible products 
and management outccmcis being generated were first identified and then 
linked to an individual capable of describing them in detail. Interviews 
were conducted with these individuals using the outputs linked to them as 
controlling themes. In the two listings or indexes which follow Table 1, 
those outputs interviewed around are annotated and each output is identi- 
fied as to its focus, i.e., research, development, diffusion, or evaluation. 
Also, each output has been arbitrarily assigned an identification number 
consisting of two parts: (£) a letter which permits easy identification 

of the output as a tangible product (P) or a management outcome (M) , (y 
a sequence number for all outputs irrespective of P or M. The numerical 
designation of the outputs appears in various tables and charts permitting 
one to examine a description and match the data from other tables to it. 
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®Full Time Equivalency. 



FIG. 2. Project organizational map. 
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TABLK 1 

Project Roster of Staff by Job Titles 



Administration 

*Project Director (TR) 
*Project Co-Director (TR) 

Clerical 

Secretary (Adams) 

Secretary (OCE) 

Technician (TR) 

Film Production (Subcontractor) 
Filming Director 
Cameraman and Film Editor 

^Formally interviewed 



Research Staff 
*OCE Coordinator 
*OCE Research Assistant 
*OCE Research Assistant 

Special Resource Person (TR) 
*Adams Coordinator 
Consultant (Adams) 

Consultant (Adams) 

Consultant (Adams) 
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Index of products. The following are the products 
the Protocol Project. 

No. Product Label 

P-01 Proposal 

P-02 Project Objectives 

P-03 PERT Chart for Project 

P-04 Work Schedule for Project Management 

P-05 Work Schedule for Conceptual Framework 

P-06 Work Schedule for Protocol Materials 
P-07 Work Schedule for User's Guide 

P-08 Task Time Line 

P-09 Articles for Teaching Research Newsletter 

P-10 Final Report 

P-11 Progress Report . The writing of a progress 

report by the staff. Provided the sponsor and 
other interested parties an opportunity to 
determine what the results of the staff's 
efforts to date had been. Summaries of work 
accomplished were submitted to the Project 
Director, and these were edited and added 
to by him. 

P-12 Cross-site Field Test Data 

P-13 National Field Test Data 

P-14 Budget 

P-18 New Proposal 

P-19 Prospectus for Refunding 

P-20 Conceptual framework 

P-21 List of Target Groups Identified 

P-22 List of Assessed Teacher Needs 

P-23 Taxonomy of Learner Outcomes . One of the 

major goals of this project is the construc- 
tion of a system of classifying learner out- 
comes. As conceived by the Protocol staff, 
effective instructional management Is a func- 
tion of matching instructional acts to 
situation-specific interaction of setting, 
learner characteristics, instructional 
strategy, and instructional resources. 

Protocol materials are Intended to produce 
specific learner outcomes. The theory for 
this model comes largely from the Special 
Resource Person. 

P-24 List of Learner Outcomes for Objectives 
P-25 Objectives for Target Population Students 




identified from 

Product Focus 

development 

development 

development 

development 

development 

development 

development 

development 

diffusion 

diffusion 

evaluation 



evaluation 

evaluation 

development 

development 

development 

development 

evaluation 

evaluation 

development 



development 

development 
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P-26 Objectives for QCE Students , This product is 
to provide a statement of objectives relative 
to the secondary education department at the 
Oregon College of Education. It will deter- 
mine the relationship of the department's 
objectives to the development of protocol 
materials and thus determine, in part, the 
priorities for the protocol films to be made. 

P-27 Objectives for High School Students 

P-28 Protocol Development Plans 

P-29 Terminal Performance Objectives for Protocols 

P-30 Protocol Specifications . In order to produce 
protocol materials, the staff is having to 
define what a "protocol" is. This involves 
determining the differences between protocol 
films and training films, thus specifying the 
determiners of a protocol and a training film. 
In addition, behavioral objectives must be 
specified around which protocols are to be 
developed, and these objectives must be trans- 
lated into learner outcomes. 



development 



development 

development 

development 

development 



P-31 Typescripts of Films 

P-32 Specifications of Filming Techniques 

P-33 Protocol Films 

P-34 Storyboards (Scenarios ) . The storyboard 
or scenario provides a map which can be 
used to produce a film. In order to write 
a storyboard, the staff members (that is, 
the two research assistants at OCE and the 
Adams coordinator) must determine the 
learning outcome(s) of the film. They 
describe situations that will appear on the 
film, particularly the specified behaviors 
that can be interpreted by the viewer as 
indicators of the concept (commitment to 
learning, constructive sense of self, 
analysis, and evaluation are four examples 
of concepts to be represented on film). 

P-35 Audio Tape Protocols 

P-36 Description of Learner Population 

P-37 Description of Context for Use of Protocols 



development 

development 

development 

development 



development 

evaluation 

development 
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P-38 Evaluation Plan * In order to evaluate the 
materials being produced, it is necessary 
to be able to define and then to measure 
the objectives of the intended instruction. 
Instruments are being devised which will 
enable the staff to make these measure- 
ments. The Project Director is taking 
the responsibility for constructing and/or 
specifying the instruments to be used in the 
preliminary field trials and the national 
operational field trials. Criteria for the 
selection of field trial sites are being 
formulated by the Project Director and his 
staff. 

P-39 Interview Strategy 
P-40 Observation Strategy 
P-41 Staff Questionnaire 
P-42 Student Questionnaire 
P-43 Tutorial Field Test Design 

P-44 Cross-site Field Test Design 
P-45 National Field Test Design 
P-46 National Field Test Site List 
P-47 User’s Manual for Protocols 
P-48 Student Information Sheets 

P-49 Student Exercise Sheets 
P-50 Advance Organizers 
P-51 Advance organizer A 
P-52 Advance Organizer B 
P-53 Advance Organizer C 

P-54 Specifications for Practicum Experience 

P-55 Student Analysis Form 

P-56 Instructor Analysis Form 

P-57 Thurs tone- type Attitude Scale 

P-58 Faculty Implementation Analysis Form 

P-59 Tutorial Field Test Data 
P-60 Conceptual Papers on Learner Outcomes and 
Human Development 
P-61 Field Test Evaluation Guide 
P-62 Instructional Memos to National Field 
Test Sites 

P-63 User Materials 

P-64 Evaluation Practice Sheet 

P-66 Description of Project . The Project Director, 
in monitoring the activities of the project 
staff, has written a description of the 
various phases of the project and has referred 
his description to others for suggestions and 



development 



development 

development 

development 

development 

development 

development 

development 

development 

development 

development 

development 

development 

development 

development 

development 

development 

development 

development 

development 

development 

evaluation 

research 

development 

diffusion 

development 

development 

evaluation 
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Index of management outcomes * The following is a listing of those 
management outcomes identified from the Protocol Project. 



No. 



Management Outcome Label 



Management Outcome Focus 



M-15 Acquisition of Equipment 

M-16 Acquisition of Operational Supplies 

M-17 Acquisition of Office Space 



development 

development 

development 



H-65 Communication with Secondary Education Depart- diffusion 

ment Chairman and Staff . By engaging in periodic 
discussions with the OCE Secondary Education 
Department Chairman and by involving the 
Secondary Education Department staff in some 
of the decision-making relative to protocol 
development, the project management intends to 
maintain a communication flow which will insure 
a high quality level of production and satis- 
factory working relationships with the college. 

M-67 Two-day Retreat . A retreat was held for the development 

purposes of clarifying the roles of personnel 
and for giving individual members a better 
understanding of the project's goals and pro- 
cedures. In addition, a review was made of the 
activities of various staff members, and an 
opportunity was provided for interaction among 
all of the personnel involved in the project. 



Output map . Figure 3 is an output map showing the dependent relation- 
ships between the outputs of the project. The relationships identified are 
not necessarily sequenced over time. 
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Chapter III; Summary of the Data 



Data were gathered around the selected outputs by means of inter- 
views with knowledgeable staff. The interviews sought to elicit for each 
output the standards by which the satisfactory completion of the output 
is judged, the tasks required to generate an output meeting those stan- 
dards, and the enablers (knowledges, skills, and sensitivities) which 
facilitate the carrying out of those tasks. Interviewee statements were 
categorized subsequently into somewhat more general statements, for pur- 
pose of providing more standardization of reported information. Tables 2-7 
summarize the data in these categories by showing how frequently an item 
of interview information was cited within one of these categories. 



Within each category are a series of descriptive labels which are 
representative of interviewee statements. These descriptive labels are 
listed in the tables under the category heading. In the process of re- 
ducing raw (interview) data, narrative interviewee statements about an 
output were linked to one of the category sets. Each narrative statement 
was then classified by means of a number code according to the most rep- 
resentative descriptive label within a given category or subcategory. 



Each table, therefore, provides the frequency with which interviewees 
cited specific statements (which are represented by the descriptive labels 
in the tables) of standards, tasks, and enablers in relation to each out- 
put.^ 



Standards Held for Outputs 



The standards cited for the adequacy of the outputs of the project 
are treated in Tables 2 and 3. In general, acceptance by others 
a higher staff level) was a criterion often used to determine whether a 
product was satisfactory or not. Because the project personnel were 
careful to aim at specified learner outcomes in producing storyboards, 
plans, and the like, the criterion of goal attainment was also mentioned 
as being important in determining the quality of an output. 



^If the reader is interested in the narrative statements of the inter- 
viewees, these can be found in the Appendix. To locate the narrative 
statement for any given category, first note the output and Its identi- 
fication number in the table. Second, note that ''' ch descriptive label 
within a given category has a distinct number or code. Turn to the 
Appendix and locate the output. Under the output locate the category 
label or heading (standard, task, or enabler) and pinpoint the number or 
numbers (depending on the frequencj cited) of the descriptive label 
which appears’ In the table. The statement in the Appendix opposite 
this number it the original narrative statement from an interviewee 
and is only represented in the tables by its descriptive category label 
and code number. 26^* 

1 s • ' 



r 



w 



T 



718 






O 

ERIC 



n 

m 

o 

o 

H 

O 

C/3 



Ln 



U> 













25 




hd 

1 


hd 

1 


hd 

1 


hd 

t 


t 


O 




I 

O' 


1 

u> 


1 

U3 


1 

N> 


I 

M 






CJN 


00 




U> 


M 


r 














03 














cr 




o 


S’ 


c/3 


H 


hd 


ro 






rt 


e 


H 


M 




CO 


03 


O 


O 






o 


M 




o 


cw 








c 




p 


H 






H* 


03 




o 


(0 








rt 


W 


B 


CO 






rt 




O 




CO 




H 


H* 


O 


03 








O 


O 


P 




o 






Ufe 


3 




O- 




(0 




(0 




hd 


co 








o 


O 


M 




r 


o 




rr 


t-h 


03 




CD 


H 








P 


c/3 


03 


rt 




o 


hd 




O 


H 






c 






(V 


3 






rt 


O 




P 


CD 






XJ 


C^t 




03 


H 






C 


(V 




H 








rt 


o 






O 






CO 


rt 




O 


c 












CO 


rt 














O 














O 














3 














(D 














CO 









M fO 



U> 



U) 



N5 K-* 



04 CoTranunlcation and clarity 
07 Personal satisfaction/feeling 

12 Goal attainment 

13 Acceptance by others (in proj) 

21 Sources of variance controlled 

22 Functions as planned 

23 Successfully constrains/guides 
30 Lack of negative feedback 



H O 
O C 
rt rt 
03 rf 
M c 
CO rr 



. 





hd 


CTi 


H 


03 

rt 


g* 


(0 


03 


oo 


H 


o 








< 


n 




03 


n 


rt 


o 


(0 


O. 


QQ 


0) 


O 




H 


P 




o 


(0 


• 


CO 


03 


o 


d 


Ml 


Cl- 


c/3 


f-* 


rt 


03 


03 


cr 


P 


0) 


CL 


f-* 


03 




H 


rh 


D- 


O 


CO 


H 


rtt 


O 


O 


o 


H 


D- 




H» 


O 


3 


c 


OQ 


rt 




T!d 


CO 


c 


(V 


rt 


rr 


CO 


U 





Output Standards Cited for Each Output Analvzed 



Process Standards Cited for Each Output 



719 



T 3 

Q) 

N 

>. 

iH 

ct 3 



o 

ERIC 



CO 

Q> 

CO 

CO 

Q) 

O 

O 

Ph 

u 

o 

CO 

T 3 

U 

ct 3 

T 3 

C 

ct 3 

4 J 

CC 



CO 

q; 

•H 

u 

o 

60 

q; 

4 J 

ct 3 

U 

CTJ 

e 

■H 

Ph 



4J CO 
:3 H 
P* CTJ 
4 J 4 J 

;3 o 
o H 



CVJ 

I 

4 J 

Q> 

CO 

60 

c 

•H 

O 

O 

P 

O 



0) 

CTJ 



T 3 

c 

CTJ 



O 

c 

q; 

o 

a 



o 

60 

Q) 

4 J 

CTJ 

u 



ro 



rjUBuraAoadun: sp^axA uopsTAa^ 

paMO^ioj aae saupx^PT^iO TV 
axqpaoAPj rjaojja jo rjoaduii 
suoT:jsanb uo paqoaaa aanso^D qz 
uo^rjop up rjpnsaa suopspoaQ 
qorjeui saAprjoafqo pue sanpp/v 9? 
uoT:jpdpop:iapd apqpssod umurpxppi qT 



CO 

p 

p 

4J 

o 



27 1 



2 



0) 

CTJ 



CO 

o 

•H 

rj 

C 

0) 

o 

c/3 



CO 

T3 

CTJ 

O 

o 



'3' 

fo 

I 

Pm 



c 

CTJ 



c 

o 

•H 

4J 

to 

o 






00 

CO 



4 J 

CTJ 

0) 

4 J 

0) 

p^ 

>•. 

CTJ 

'ca 

I 



CO 

H 

o 

H 

p^ 

o 

o 

H 



vO 



720 



Tasks Peru ilnlng to Output Attainment 

Prominent among the types of tasks (Table 4) required in generating 
outputs were identifying and clarifying the problem, formulating objec- 
tives, and actually producing the output. In producing plans for evalu-- 
ating the protocols, designing was the most frequently cited task. 
Identifying and clarifying the problems involved in devising plans was 
also mentioned. 



Enablers Pertaining to Output Attainment 

Among the knowledges (Table 5) required to perform the tasks, 
information outside the project context was cited more frequently than 
was either information generated within the project or knowledge of 
technical subjects. The nature of the project is such that considerable 
knowledge of classroom situations and of children is necessary in order 
that realistic scenes be presented on film. Accordingly, the experiences 
of the staff members have proved to be Invaluable to them in planning 
film sequences which can present tht four concepts of commitment to 
learning, constructive sense of self, £malysis, and evaluation (the last 
two from Bloom's Taxonomy of Educational Objectives ). 








Tasks Cited for Each Output Analyzed 
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Chapter IV : Supplementary Data 



Additional data with respect to this site visit come from ques- 
tionnaires obtained from the interviewees as well as classifications 
made of the outputs -o£. .the project. This chapter also includes a 
summary of staff backgrounds and requirements of the various staff 
positions. 



Classifications of Output Characteristics 



As the Oregon Studies evolved it became evident that outputs 
could be categorized in terms of a number of variables. Among them 
are (^) Structure (product, event, or condition), (^) Function 
(policy setting, management, or production), (O Level (focal, com- 
ponent, or Jacilitating) , (^) Character (knowledge, technology, 
implementation, or information), and (^) Stage of completion* These 
five schema are represented in Table 8 for each project output iden- 
tified, with frequencies summarized for each category. Table 8 has 
been added to this profile subsequent to the profile's original 
writing. 



Summary of Staff Backgrounds 

Personnel data were obtained from six of the staff members. Of 
these, three hold doctoral degrees, two hold master's degrees, and one 
has a bachelor's degree.^ Major areas of specialization include educa- 
tional psychology (2), educational administration (2), anthropology, and 
education/ teaching. The two co-directors of the project are full-time 
research workers, whereas the others working on the project, with the 
exception of the OCE Coordinator, devote only part (i*e«, half time or 
less'^'-to research activitie 

The work experience of the interviewees has been largely in teaching. 
With the exception of the Director and the OCE Coordinator, the major 
type of work experience of the interviewees has been public school 
teaching, with a range of 9 to 20 years. Three of the interviewees 
have had no previous experience in research. The three administrators 
had been engaged in research activities from 2 to 7 years. Two of the 
interviewees have been or are college teachers. 
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The most frequently cited aspect of professional training considered 
relevant to this project was the course wot c taken by the interviewees in 
college. Psychology, sociology, anthropology, statistics, measurement, 
and English were mentioned as courses which have proved helpful in 
connection with the work of this project. Also prominently mentioned was 
on-the-job trainin^g in research, specifically research concerning instruc- 
tional media anc^ j^urriculum construction. Writing experience was considered 
as very important by one interviewee. 



Summary of Position Kegu Lrements cjnd Support System 

The Director and Co-director of the project responded to questions 
concerning the requirements for their positions in similar ways, empha- 
sizing experience in instructional development and research experience 
(the management, organization, and evaluation of research efforts). The 
Co-director, who has had considerable teaching experience, also cited 
teacher education and teaching experience as requirements for the position. 
The two coordinators who responded emphasized learning theory as important 
in their jobs, one citing a knowledge of preadolescent and adolescent 
psychology as a requisite. Tlie two research assistants cited the same 
kinds of requirements as did the directors and coordinators, adding 
experience in film writing (which was their major task) and statistics. 

Among the services needed for this project are duplicating and dis- 
tributing systems, secretarial services, typing, library services 
(including requests for documents), photography, statistical computation, 
accounting, audio-visual aids (including television facilities) , film 
editing, and subjects for experimentation. By virtue of the nature of 
the project, many different kinds of educational media services are needed. 

Summary of Inservice Training Experiences of Staff 

Because the Protocol Project is one of a dozen such projects which 
have been funded throughout the country, training institutes have been 
held for the directors of the projects, with additional institutes 
scheduled during the remainder of the academic year. In addition, mem- 
bers of this project held a retreat which has served to train and acquaint 
all of the staff in the theoretical and practical issues which are being 
encountered. Frequent staff meetings also serve to brief members on 
developments and to focus attention on current and future problems. 
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Chapter V: Project Dynamics 



Leadership of the Protocol Project was centered in three people. 

First, t!ie Spec;,ial Resource Person was instrumental in originally concep- 
tualizing tlie project, writing the proposal, and obtaining funding. He 
Iiolds the academic rank of "professor" and devotes 20% of his time to this 
project. The remaining 80% of his time is devoted to directing a ’^program*' 
at Teaching Research which consists of several projects in addition to 
the Protocol Project. In effect, he contributed intellectual and conceptual 
leadership to the project. The second point of leadership was in the con- 
tribution of the Project Director. Devoting 40% of his time to this project, 
his primary contribution was seen to be the planning and guiding of overall 
project efforts and promoting Interagency coordination. The third person 
charged with a primary leadership role was the Co-Director. She, like the 
Director, devotes ,40 FTE to the Protocol Project. This person prlmariiv 
assumed the role of project manager and was concerned with work-scheduling, 
monitoring, and training. 

Within each of the two other collaborating institutions (Oregon College 
of Education and John Adams High School) another leadership role was seen in 
a "coordinator" for the efforts of each of these agencies. While the Adams 
coordinator contributed his full time (1.0 FTE) to this project, the OCE 
Coordinator carried teaching durles allowing only .25 FTE for the Protocol 
Project. The OCE coordinator guided the production efforts of two Research 
Assistants. The Adams coordinator assumed the production role, utilizing 
various consultants within his organization to assist his efforts. 

Figure 2 (Chapter II) graphically displays the organizational pattern 
described above. Several of the interviewees expressed the feeling that 
the project had "lots of chiefs but few Indians." Within the Teaching 
Research portion of the organizational chart, the title of "Co-Director" 
may appear to be a misnomer. Although the Director and Co-Director were 
assigned to the project with equal FTE (.40), and were considered of equal 
status by the parent agency, the interviewees generally appeared to consider 
the Director as the primary leader of the project and the Co-Director as 
the manager. The tasks performed by these individuals appeared to support 
this view of their activities and relationships. 

The staffs of OCE and Adams High School were relatively inexperlc need 
in the area of materials development. Although each of the staff members 
had experience in the utilization of various audio-visual teaching materials, 
they had no knowledge of production techniques. Actual productl,:n was to be 
accomplished by a subcontractor, but the preparation of storybor.rds and scripts 
for guiding production required knowledg'j beyond that possessed by the proj- 
ect’s writers. This deficiency required an unexpected effort to be made in 
training. 

The necessity for training and «:ubsequent follow-up monitoring of work 
efforts made almost daily communication iiecessary. In this area additional 
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unexpected difficulties were encountered. Since the two OCR RevSearch 
Assistants were half-time students, their availability for meetings was 
somewhat curtailed. In addition, their offices were not equippf^d witli 
telephones which imposed a burden on clerical personnel outside of the 
project. The Adams High School coordinator had only part-time clerical 
assistance from the school’s regular clerical personnel. His location 
(65 miles away) precluded easy access to the Teaching Research facilities, 
and travel time to Teaching Research made his attendance at frequent 
meetings difficult. To offset some of these problems, a retreat was 
planned so that all personnel could more easily coordinate their efforts. 

The interviewees from each of the collaborating agencies indicated 
that each institution held differing, although noc necessarily conflicting, 
objectives and goals for their participation l.n the project. Teaching 
Research, as holder of the contract for the project, saw the project 
producing exemplary protocol materials, establishing a prototype set of 
procedures for the development and testing of such materials, and pro- 
viding an opportunity to carry out basic conceptual work in regard to 
classes of learner outcomes. They also saw the project as a vehicle to 
further cooperative relationships with Oregon College of Education and 
John Adams High School. 

OCE and Adams High School saw the project primarily as a means of 
producing materials that would be of value in their own teacher training 
programs. Since the particular focus of the OCE desire was for materials 
in support of secondary teacher education at the preservice level and the 
particular focus of the Adams desire was for materials In support of sec- 
ondary teacher education at the inservlce level, the interests and the con- 
ceptual approaches of these two institutions were somewhat diverse. 

The differing viewpoints of the three, cooperating organizations have 
caused special effort to be made on the part of Teaching Research to maintain 
a common focus in this developmental project. The degree to which Teaching 
Research is successful in this effort will likely depend upon the degree to 
which the people from each institution begin to see that their differing 
needs or interests are being met. 
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Chapter VI: Implications for Training 



This chapter summarizes the data obtained from the interviewees 
regarding training needs for professionals in educational research, 
deve lopment , .di f fusion , and/or evalviatLon. 

In several ways the Protocol Project is typical of projects in whicli 
people must be trained on the job. At the inception of this project 
no one was certain — not even the advisors to the 12 protocol materials 
projects throughout the country — what protocol materials would look like 
when they were produced. Therefore, in a real sense, everyone on the 
project is being trained on the job. 

A background in film making for project personnel is obviously 
needed if the decision is made to use the film medium for protocol mater- 
ials. In addition, since the project is concerned with developing a 
taxonomy of learner outcomes and accompanying ancillary materials, 
experience in writing curriculum materials is highly desirable. 

Both practical experience and academic preparation were cited by 
the interviewees as being valuable to staff members. The Co-director 
indicated that course work in psychology, sociology, anthropology, and 
English was most important in preparing for the project duties of that 
position. The OCE Coordinator indicated as important academic prepara- 
tion in statistics, experimental design and measurement. 

In general, the administrators who held doctorates emphasized the 
importance of having directed other projects and of having the proper 
academic preparation for their positions on the project. The research 
assistants, on the other hand, naturally felt that both practical 
experience in the classroom and a genuine understanding of children and 
teaching problems were necessary in carrying out their functions. Both 
the administrators and assistants agreed that communication skills 
are important to persons who engage in projects of this nature. In his 
response to questions concerning training for jobs such as his, one ox 
the research assistants emphasized personal qualities such as open- 
mindedness and imaginativeness. He also mentioned a quality that probably 
makes all the difference in any project, namely, a belief in the worthi- 
ness of the undertaking (and with it, perhaps, dedication to a cau'"^' . 
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Appendix: listing of Output Standards, Tasks, and Enablers 

The following is a list of standards, tasks, and enablers for outputs 
around which interviews were conducted. These statements were extracted 
from discussions with ..uterviewees and were coded into their respective 
category sets. The c elected code precedes the statement and indicates 
the following for: 

STANDARDS 



Code J : 


Structure of Stuaaards. 


J-l 


Standards against which outputs are judged. (output 
oriented) 


J-2 


Standards against which processes and^c^r operations are 
judged, (process oriented) 


Code LM: 


Primary Categories o^ Standards. 


TASKS 
Code NO: 


Clusters of Tasks. 


ENABLERS 
Code S: 


Structure of Enablers. 


S-1 


Knowledge. 


S-2 


Skill or ability to perform. 


S-3 


Sensitivity or awareness. 


Code UV: 


Primary Categories of Enablers, (knowledges, skills, or 
sensitivities) 



The codes associated with these three categories (standards, tbsks, 
or enablers) are the same both here in the listing and as previously cited 
in Chapter 111 tables. 

Each of the nine analyzed outputs is cited below In a rectangulai.* '^ox. 
Included within each box is the level of the output (l«e«, focal, component, 
or facilitating) and the status of the output at the time of interview. 

Listed under each box are the interviewee statements relevant to that output. 



P-11 : 
Level : 
Status : 



Progress Report 

Focal 

Ongoing 



/ J j 



STANDARDS: 

1 M 

1 13 Director accepts report (with only minimal changes). 

TASKS : 

NO 

01 Jtuay format (topic outline; of report (supplied oy Director). 
22 Direct staff to submit summarijs of work done. 

05 Extract the essential items froa staff summaries. 

04 vVrite report in rough draft form. 

06 Edit draft of report into final format for rough typing. 

24 fubmit rough-typed report to Director for review. 

22 Direct secretary to type final copy of report. 

06 Proof-read final typing. 

22 Direct secretary to duplicate and mail report. 

ENARLERS : 

S W 

1 01 Knowledge of basic English and grammar and vocabulary. 

2 06 Skill in logical organization (of budget categories) — ’’Looking 

at it as a reader might.” 

2 17 Skill in recognizing jargon. 

2 32 Skill in editing other people's copy. 



P-23: Taxonomy of Learner Outcomes 

Level: Component 

Status: In Process 



STAI^DARDS : 

J 

1 12 Groups of behavior are generated which are amenable to 
protocol development . ^ 

1 12 Protocols (or behaviors) could be accounted for and classified 
by the taxonomy . 

. : 
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TASKS: 

m 

31 Participated in discussions dealing with issues relative to 
the development of taxonomical schema. 

02 jetermine how behavior can be classified. 

03 Specify the broad categories of behavior . 

03 Attempt to understand how the taxonomy could fit .'.nto the 
developnent of spec if ic protocol materials . 

ENABLERS: 

1 jiv 

L 03 Knowledge of whnt has been done in the field relative to 
taxonomies (in education) , 

1 04 Knowledge of human behavior. 



P-26: 


Objectives for OCE Students 


Level : 


Component 


Status : 


Completed 



STANDARDS: 



No information collected under this heading. 



TASKS : 

NO 

29 Provide statement of objectives relative to the secondary 
education department (at a teachers* college). 

05 Rank ordered the (above) objectives. 

03 Determine relationship of department’s objectives to the 
development of protocol materials. 

01 Determine priorities of protocol development. 

ENABLERS: 

S 

1 23 Knowledge of student response and needs in a (teacher) 
training program. 

1 25 Knowledge of what is available in terms of materials. 

1 06 Knowledge of where you are going (secondary education program). 

1 06 Knowledge of (department’s) needs relative to instructional 
materials « ^ 
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1 (j 6 Kncviedge (based on experience) ot what seems to be nuccesHfol/ 
unsuccessful in terms of student performance and efficacy of 
Instruction — referring to mode of instruction, such as media 
which in turn links to type of needed instructional materials. 

1 19 Ability to determine how a protocol will fulfill tho.se needs. 

2 24 Ability to determine department needs in termvS of the secondary 

education program. 



P-30: 


Protocol Specif icat ions 


Level : 


FaciJ itating 


Statui : 


Completed 



STANDARDS: 

No information collected under this heading. 



TASKS: 

01 Determine the meaning of a '‘protocol.** 

01 Determine the differences between a protocol and a training 
film. 

01 Specify the determiners of a protocol. 

01 Specify the determiners of a training film. 

04 Prioritize Bloom's taxonomy in terms of protocols that are to 
be developed relative to a specific objective or level. 

02 Select an objective (or objectives) around which a protocol 
is to be developed, 

01 Consider time and budgetary constraints in the number of 
objectives to be selected for protocol development. 

03 Translate objectives into learner outcomes. 

03 List ways in which one could evidence those learning outcomes. 

03 Determine whether protocol film should be process or product 

oriented . 

ENABLERS : 

1 02 Knowl«‘dge about instructional techniques in education 

1 04 Knowledge as to how learning occurs. 

1 04 Knowledge of human behavior, 

1 06 Experience/background in classroom instruction, 

1 08 Knowledge of determiners of what constitutes a protocol. 
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P-34: 


Storyboards (Scenarios ) 


I.evel : 


Fad li tat ln% 


Status : 


Completed 



STANDARDS: 

1 Ml 

2 26 Apparent mutual undei standing of direction of effort is achieved. 

2 42 Kach rewrite of scenario improves it. 

1 22 Scenario conveys intended education message. 

1 23 Scenario effectively guides production. 

1 12 Checklist showed students in the protocol were successful in 
their analyzing. 

1 13 College coordinator approved the storyboard idea. 

1 23 On the films the students were doing what they were supposed 
to be doing (such as analyzing). 

1 23 The students were saying *'what they were supposed to say'* 
(according to a prepared checklist). 

1 12 Components of a storyboard appropriately represent the 
specif led concept . 

1 04 Relatively "uncluttered" from a lot of extraneous ideas. 

1 13 College Coordinator approved the idea for a storyboard. 

1 13 Co-Director approved the rough script (of a storyboard). 

1 30 "When they don't send it back and say, 'Do it againi'" 

1 07 "I feel that it's pretty well ready to go." 

2 34 When someone says, "it's coming along." 

TASKS : 

NO 

01 Read literature relative to the subject matter. 

02 Read proposals (for protocols) from other organizations. 

01 Make notes of important items found from study* 

01 Read assigned material with associate. 

02 Discuss assigned materials with associate. 

31 Call and participate in staff meeting. 

29 Interpret background material to achieve mutual understandf . 

31 Participate in open discussion to achieve clarity and under- 
standing. 

22 Schedule work sessions witli junior staff. 

31 Conduct work sessions with junior staff. 

26 Interact with junior staff to guide their efforts. 

21 Teach junior staff to use consistent terminology. 

24 Read scenarios which were written by the staff. 

06 Note advisable corrections to scenarios. 

31 Discuss recommended changes with writer. 

22 Direct writer (junior staff) to rewrite. 

30 Act. as catalyst to discussion to bring Issues to foreground. 

02 Come up with ideas for the content of final protocol films. 

01 Examine ways Bloom's Taxonomy of Educational Objectives could 

serve as a basis for content of the protocol. 

" 28a 
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Oj Determine what elements are to be used In the analysis 
activity lo be used In the protocol. 

03 Determine how the elements relate to one another. 

02 Confer with high school coordinator about scenario ideas. 

02 Confer with fellow research assistant about scenario Ideas 
that are generated. 

02 Confer with Co-Director about "he scenario Ideas. 

01 Do library research about how idvertlsers try to sell rhelr 
products . 

04 Write Indicators (that lS| responjcs of students) which would 
show that the students In the film were demonstrating the 
ability to evaluate. 

29 Inform teachers (In scenarios) about the objectives of the film. 

04 Write Ideas that the (scenario) teacher might elicit from his 
students (on the film). 

29 Discuss with the television teacher background information 
concerning the concepts presented. 

29 Discuss the Ideas and activities to be filmed with the student. 

02 Talk with the college coordinator and project directors about 
the storyboards. 

04 Outline the composition/content of a storyboard. 

03 Determine specific examples representative of the objective. 

04 Specify the objective — which Is a component of the storyboard. 

04 Specify the learning outcomes — which Is a component of the 

storyboard. 

02 Descr, je the background Information for the situation that 
will appear on the protocol film which Is a component of the 
storyboard. 

04 Describe the situation that will appear on the film which Is 
a component of the storyboard. 

04 Describe the behaviors of kids that will be accepted as 
evidence that the objective has been achieved which Is a 
component of the storyboard. 

03 Determine the purpose of the storyboard — describe the situation 
and Intent of the protocol film; serves as a basis for the 
production of films. 

04 Develop a mock-up/prototype storyboard. 

24 Rev lew/ critique storyboard. 

24 Monitor the work/production of a storyboard. 

31 Answer questlons/asslst In solving problems relative to the 
development of the storyboard. 

01 Identify the problems for storyboards. 

01 Evaluate what has been done In terms of the concept we are 
trying to illustrate. 

01 Consult curriculum guides and teacher manuals for Ideas for 
protocol films . 

02 Consult persons In field, such as teachers, for Ideas about 
protocol films. 

03 Determine the process (e.g., analysis) to be Illustrated on 
film. 
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CoiiHul t with cowor xpr concerning storyboard ideas . 

Consult with college coordinator about materials which might 
be used. 

02 Consult ^rlth college coordinator about storyboard Ideas. 

01 Head bloom's laxonomy of Fxlucatlonal 0bl<»ctive8 . 

01 Head a variety of materials for Ideas for storyboards. 

KNAliLEKS; 

S ^ 

1 01 Know normal school subjects (high school). 

1 OA Know enough about the subject to see relationships even If 

the author's words are different. 

1 04 Know learning theory. 

1 06 Know normal classroom activities. 

1 00 Know normal classroom appearances. 

1 06 Know normal student reactions. 

1 08 Know the purpose for which you are r^.adlng. 

1 08 Know project objectives. 

1 24 Know tasks that others must do. 

1 24 Know housekeeping details (where to get t/pe). 

2 02 Able to Interact effectively face to face with junior staff. 

2 02 Serving as a catalyst without hurtlag either party. 

2 08 Ability to sift and sort and evaluate. 

2 08 Skill in analytical reading. 

2 08 Skill In analytical thinking. 

2 09 Recognize need for acting as a catalyst. 

2 09 Able to exercise patience In working with junior staff. 

2 11 Able to accept Junior staff capabllftles (do not dowm grade) 

2 17 Translating Information Into project terms. 

2 17 Translating progrsslonal shorthand (jargon) Into terms under- 

stood by junior staff. 

2 19 Sense of organization of work effort. 

2 46 Reassuring staff In face of apparent ambiguity. 

3 02 Aware of content In which colleagues work. 

3 03 Sensitive to junior staff needs (materials). 

3 12 Recognizing the effective time to act as catalyst. 

3 13 Attention to vocabulary as clues to misunderstanding. 

3 18 Alert to trouble spots before they gel bad. 

3 49 Willingness to learn from others (authors, etc.). 

3 50 Intellectual openness In dealing with staff. 

3 51 Alert to junior staff points of confusion. 

3 51 Alert to cues that junior staff Is confused. 

3 32 Attention to affect (junior staff effort). 

3 52 Aware of Impact of ambiguity (on staff). 

1 06 Familiarity with what goes on In school and "the total teaching 
situation. " 

1 06 Understanding what children can do (academically). 

2 Imagination of concepts which can be taught on film. 

2 14 Wrltl ig ability to express concepts on the scenarios. 

2 02 Ability to "work with kids." 
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J. 19 
i 0? 
I lA 
L 0 ’ 

1 3A 

2 02 

j 30 

3 01 

3 30 
3 53 
1 02 
1 08 
1 08 
1 08 

2 01 

2 03 

2 19 
1 01 
1 06 
1 06 
1 01 
1 02 
2 lA 
2 17 

2 65 

3 35 



Ability to organize details. 

Ability to communicate with other people on the project. 

Writing ability. 

Ability to work with students. 

Being able to take care of details. 

Ability to conununlcnte with coworkers (being able to underst.mil 
what the other person is saying). 

Understanding about perceptions and fncllngs of young I'cople 
(trainees) . 

Understanding that everyone has his biases and these h.nve 
"to be put In the table and recognized." 

Sensitive to children's feelings. 

Imagination — "a bit of creativity." 

Understand Bloom's taxonomy. 

Know the purpose of a storyboard. 

Know what the project Ir at out — Its purpose. 

Knowledge of mental processes — that they do in fact exist, such 
as "analysis." 

Ability to communicate what a storyboard might look like for a 
particular objective or concept. 

Ability to wrlce/develop a model to aid in the understanding 
of the process of st'^ryboard development. 

Ability to conceptualize what a storyboard might look like. 
Knowledge of subject matter of film (e.g. , math or advertising). 
Knowledge derived from classroom experience. 

Experience as an actor. 

Knowledge of literature and/or drama. 

Knowledge of cognitive levels (Bloom's Taxonomy). 

Skill in script writing. 

Translate what is written In curriculum guides to a classroom 
situation. 

Skill in doing library research. 

Be sensitive to the lines of communication being open. 



P-38 : 


Evaluation Plan 


Level : 


Facilitating 


Status : 


Completed 



STANDARDS : 

J 

2 A1 Feel sure that the evaluation will get the data wanted because 
the ideas were derived from the consultant's Ideas us 
expressed in written documents. 

■4OA 

, 40 , 



J 



1 12 Data collprtef'i helfq make doclalonn. 

1 21 Instruments used have underro<ie -is mnnv ar seven validations In a 
previous Btiulv. 

TASKS t 

Ol knmillarlxc self wltl. project objectives. 

01 Familiarize self with objectives of instruction as defined v/lthln 

the project. 

01 Identlfv related Icarnlnf; svstom characteristics for measurement 
from experience on lov-cost project. 

03 Tdentllv Instruments to measure objectives of Instruction 
and system characteristics. 

01 Define terms In contractual commitment (such as "general izabllltv") 
ns system characteristics to allow measurement . . 

04 Modify Instruments to fit the project evaluation measures. 

03 Specify a three phase evaluation plan from the instructional 

development system developed In another project: (£) Trvout and 

revision, (y preliminary field trial, (c) national operational 
field trial. 

04 Outline trvout and revision procedures in inforiral meetings with 
staff 

05 irvout the protocol tapes to assess the need for revision. 

02 Discuss any suggested changes with staff. 

23 Select diverse (In Instructional viewpoint) Institutions (a college 
and a high schoil) to develop and test each others materials. 

0.3 Speclfv exchange of materials and data collection between a college 
and a high school as the preliminary field trial x;hich 
was to measure generallzability . 

03 Specify the instruments to be used in both the preliminary 
field trial and the national operational instruments. 

03 Specify acceptability levels of measurement for the instruments. 

03 Write one paragraph procedure star»ment for each instrument. 

03 specify that the national operational field trial: (y be ad- 

ministered by mall, (b) that materials be sufficient for self 
administration, (c) thar developers of the protocols not be 
involved In the field trials themselves, and (y that test sites 
try both sets of protocols both as suggested and in any means 
they chose so long as they explain how they used thtan. 

23 Meet with staff to determine the number of and the institutl” 
to participate in the national field trial given financial and 
staffing constraints. 

03 Discuss criteria for selection of sites based on experience from 
low-cost project. 

31 Write weekly memos to participating sites to inform their repres- 
entatives of (yaspects of the project with which they should be 
familiar, (b) other sites participating in the trial, and (y 
responsibilities in carrying out the field trial in the form of a 
sequencial check-list. 
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Ocsign the memos to stand out and call attention so that they 
will not he forgotten or misplaced. 

Specify Interviews as a final data collection measure to col- 
lect all other relevant data. 

Select a set of Questions for the interviews from experience 
with possible questions sets. 



Famlliarltv with goals of project and materials, the constraints 
In litcratvire sources, .and with the request for proposal. 

Know what, why, and how to evaluate the matcri.Tls. 

Familiarity with models which measure what, why, or how evalua- 
tion . 

Familiarity with literature sources on evaluation design. 
Familiarity with articles on evaluation. 

Knowledge of instrument selection and fabrication. 

Able to relate specific objectives to the goals of the project. 
Able to measure how the svstem is used, i.e., implementation, 
analvsis. 

Skill in citing procedures. 

Sensitive to how much a student will take before he says, 

"You're evaluating me more than you're teaching me." 

Sensitive to manageabilitv of data, i.c«, collect only as much 
as you can analyze. 

Sensitive to what faculty will stand for in terms of mount 
of reading and time required. 

Sensitive to the goals of the project. 



M-65s Communication with Secondarv Education Department 
Chairman and Staff 
Level! Facilitating 
Status: Ongoing 



No information collected under this heading. 
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STANDARDS : 



IU'3 



TASKS: 

31 Enpage In periodic dlflcuBslons wltli Sccondarv Education Depart- 
ment Chairman to dlscuBR proprenB and/or stntUR o' the project. 

33 Attempt to Involve Secondary Education Department staff In 

some of the decision making relative to protocol development. 

24 Ensure protocols reflect the objectives of the Secondarv 

Education Department by periodic discussions with the depart- 
ment's staff. 



ENAIiLERS: 

S W 

3 02 Sensitive to the amount of information that Is to be presented 
to a person 

3 02 Consideration of another person's time or schedule. 

3 04 Sensitive to the nonverbal cues which may be transmitted by 

the person(s) to whom you are talking. 

3 11 Sensitive as to the directness with which you deal with the 
Information and present it. 

3 27 Sensitive to the timing of the irformatlon presentation. 



P--66: 


Eescrlntion of Project 


Level : 


Component 


Status : 


Onp^olnjB; 



STANDARDS : 

J 1^ 

1 07 Author feels satisfied with what has been written. 

TASKS: 

rra 

04 Write descriptions of the various phases of the project. 
24 Subject writings to scrutiny of other project and depart- 
ment staff for suggestions and comments. 

ENABLERS : 

S W 

2 04 Ability to Interpret and 
have occurred. 

2 14 Ability to write clearly 



ERIC 



make some analysis of things that 
and succinctly. 
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2 14 
2 18 


Technl'.al aklll of writing and presentation. 

Ability to evaluate the worth of eotnetbing in terms of 
questions like productivity; value to the department, etc. 


218 


Ability to identify Important issues*— and recognize their 
importance. 


2 18 


Ability to .judge whether something 1s reallv worth pursuing 
and discussing or if it is inconsequential. 


2 47 


Mental skill recall. 



M-67: 
Level : 
Status : 


Two-day Retreat 

Facilitating 

Completed 



STANDARDS : 



J LM 
2 10 


Problems concerning role, responsibility, and decision 
making are openly discussed. 


2 28 


Individual staff members indicate verbally a clearer under- 


2 27 


standing of the project goals and procedures. 

Decisions reached in the meetings result in operational plans 
for continuing the work of the project as efflclentlv as 
possible. 


TASK.S : 
NO 
29 


Resolve confusion in project staff concerning responsibility 
and declcion making by group discussion. 


29 

29 


Clarify uroject tasks for the staff by group discussion. 
Refine Indicators of learner outcomes by pooling of ideas 
in group dlrcusslon. 


29 


Review what staff members are doing in project, in group, 
to pronote communication. 


25 


Clfrlty for particular staf^ nembers that have caused some 
problems in the work of the project, what the source of those 
problems are and how they can be resolved. 


31 


Present personal project work (storyboards ^ to the whole staff 
in group and defend work conceptually. 


29 


Interact with other staff to clarlfv concepts and personnel 
problems . 



ENABLERS ? 



S UV 
1 28 


Knowledge of the basic principles of group dynamics. 
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2 35 
2 29 

2 09 

3 13 
3 22 

3 04 



Able to use words effectively with Individuals in a group, 
Able to guide an individual, usually in a groun, baeV to 
the task at hand when he or she starts in a direction ‘hat 
is inappropriate to the task. 

Able to synthesize verbal Input from many people in the gioup 
to articulate group goals and ob.lectlvcs. 

Sensitive to what certain words mean to different people. 
Sensitive to the structured or unstructured approaches that 
different people take to a group process lll;e a retreat. 
Senaltlve to how ovm behavior effects otherfs in the group, 
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A PROJECT OF: Westinghouse Learning Corporation 

2083 West Street 
Annapolis, Maryland 21401 



O 

ERIC 



29o 



f 



w 



750 



I 

I 



► 










This profile has been prepared according Co 






PROFILE FORMAT No. 1 






Three profile formats are represented in this volume. 
The reader should refer to this number when making 
use of che reader's GUIDE to the profiles. 
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Chapter I : Overviow 



This chapter contains a narrative introduction to the Annapolis 
Project, Including the objectives, rationale, and significance of the 
project and the context in which it operates. 



Synopsis of the Projec t 

Title: Development of a Multimedia Course in Leadership for 

the U.S. Naval Academy. 

Responsible Institi "'ion : Westinghouse Learning Corporation. 

Funding Source: 1. U.S. Office of Education. 

2 . Westinghouse Learning Corporation . 

Funding Duration: June 28, 1968 to June 26, 1971. (36 months) 

Observation Date: January 1971. 

Present Stage of Development: Near final in course development; 

midway in research aspects. 

RDD&E Focus of Project: Dual foci; Educational research and 

development, with a strong secondary focus in evaluation, 
coupled with the primary or research focus. 

Expected i. a multimedia course in Leadership for the 

Outcomes: U.S. Naval Academy. 

2. Extended knowledge about learning from the 
media used, obtained from the research. 

Level of Funding and Duration: Medium-High. (level 4 of 7 levels) 

Agency Setting: Industry-sponsored instructional materials 

corporation . 

Setting of Primary Location of Work Efforts: University. (Navt 

Academy) 



Staff Summary (current) : 

Staffing on this project has fluctuated widely over the project's 
duration; thus, total full time equivalence in wan years is 
difficult to compute. Personnel, both professional and support, 
are full time for the duration of their employment on the 
project . 

The number of staff formally interviewed was 12, with morr 
being interviewed informally. The latter interviews obtained 
little hard data, but did provide some of the contextual 
data that is utilized in this case profile. 
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nhj ec 1 1 v(\ s I R ationale y an d Sign if i cancc the Project 

ObJ ecti vcs » '1 h is proj oct has two p rimary thrusts : 

J. The development and production of a multimedia course In 
Leadership, Psychology and Management for the Naval Academy, as 
an improvement on the traditionally taught course. This change 
is one for which the Academy felt a need. 

2. Research into the relative merits of various media available 
for various subject content handling, and into the relative impor- 
tance of different variables connected with tiie use of various 
media . 

The Academy benefits from the pioject by receiving the improved 
material, while the Corporation receives the benefits of the research 
findings . 

Rationale . To correspond with the two primary thrusts of the 
project, each has a rationale: 

1. The Navy has for many years been a pace setter in the adoption 
of electronic equipment as aids to navigation, fire control, and 
other operations. Consistent with this, it seemed appropriate 
that the course in Leadership, Psychology and Management should 
avail itself of whatever equipment best suits the purpose of the 
course through multimedia use. 

2. Use of varied media facilitates research into the nature of 
learning and its dependence on variables that would be much more 
difficult if not impossible to measure under any other circumstances. 
The results of this research should prove invaluable in designing 
future multimedia courses in various curriculum subject areas. 

Significance ot the project . The plan is unique both in the number 
of factors investigated and in the use of an entire ongoing course system 
as an experimental vehicle. This permits empirical findings to be extracted 
relevant to the influence of each factor, and of factors in combination. It 
is expected that experiments of this type, as part of a concentrated 
effort in educational research, may eventually result in a comprehensive 
understanding of the educational process, so that an instructor may 
choose with confidence those media and presentation forms most effective 
for each particular type of student and particular type of task. 



Context in Which the Project Operates 

R elationship to parent agency . The division for research and develop- 
ment of the patent agency established an office for this project in 
close proximity to the Academy that constituted the immediate user 
agency (see Figure 1) . 
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U.S. Office of Education 




FIG. 1. Contextual map. 
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Relationship to funding sponsor . The essential contract, from the 
project viewpoint, is with the immediate user agency, the Academy, The 
project is funded by the U.S# Office of Education (USOE), through the 
National Center for Education Research and Development (NCERD), Dis- 
bursal of funds is subject to approval by the user agency of the course 
material delivered by the project. 

# 

Relationship to immediate user agency . While the project is respon- 
sible for producing all the pieces and parts of the multimedia course, 
every piece and part must meet the approval of the Academy. Such approval 
is sought at virtually every stage of production. 

Relationship to subcontractors . Certain products, both textual and 
machine dependent, have required extra help beyond the capacity of project 
personnel. This work has been contracted out. Such contracts are nego- 
tiated by the project personnel responsible for the immediate product 
concerned . 

Relationship to advisory group . A university consultant group is 
retained by the Academy (the user agency), and is responsible for advis- 
ing the Academy in matters relative to the overall program which contains 
this project. 

Relationship to other efforts of an overall program . The project 
described here is one of three connected with the same user agency. The 
other projects were committed to produce courses in different subject 
areas, covering a range from **hard” to *'soft.” The agency selected to 
conduct each project was responsible for the planning and development of 
its own commitment, with the same close approval of steps by the Academy 
and/or its advisors. 



Supporting and Technological Resources 



Provided by parent agency . Apart from the in-office facilities pro- 
vided at the location opened for the conduct of this project, the 
principal support is a remote computer facility providing computer- 
managed (CMI) data handling for the project. 



Provided by user agency . The Academy provides instructors, class- 
rooms — in which the classes for which the course is being developed 
would normally be conducted — and computer-aided instruction (CAI) 
terminals in individual cubicals. These CAI units are available for 
this and other courses beyond the immediate group mentioned in this 
overall program. 

Provided by subcontractors . Two types of resource are provided 
by subcontractors: 



1. Additional writers with expertise in/specific subject matter 
areas . 

O 



I u 
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2. Equipment, such as for making video tapes, movie films, or 
for dupl-^atlng tapes. Printing facilities also fall within 
this group of resources. 



Physlcal/Environmental Setting 

Work location. The project offices in which the production work 
takes place are located in a modern building across town from the Academy, 
making each readily accessible to the other without being in continuous 
contact. This was an arrangement seen as ideal by the project management: 
neither too close, nor too remote. 

Availability of resources . Any resources needed by the project are 
located in the same city as the Academy, or in other major East Coast 
cities within easy distance. 

Time line . A time line on a project as complex as this would be 
extremely difficult to display. Figure 2 shows the general sequence that 
typical elements of the different products must past through, as well as 
the associated schedule of technical and milestone reports. However, as 
this process is ongoing throughout a 12-part course, assignment of specific 
times would involve too great a complexity of dates. 
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Chapter II: Parameters of the Project 



This chapter discusses t)ie functional structure of tlio Annapolis 
Project and describes the products being generated. 



Project Structure 

Staff structure . Figure 3 graphically illustrates the staffing 
structure of the project. Somewhat unique to this project is the problem 
of defining its boundaries. Normally the staff would be those people 
employed on the project within the agency to whom the contract for the 
project is granted. In this instance, the close working relationship 
between the project and the user agency in bringing about the products 
of the project would make strict limitation at such a boundary difficult 



While there is a very definite and, at times, sharply drawn distinction 
between the agencies represented to the left and right of the chain-dotted 
line in Figure 3, product generation involves interaction between them 
such that the boundary seems to disappear. 

The staff member designated "instructor** has worked at the Academy 
in that capacity for the purpose of becoming fully familiar with course 
operation, and continues to work there for implementation of the new 
materials in instructional use. 

The two staff members described as "course coordinators*' are both 
employed by the project agency, but one works at the Academy and the 
other at project headquarters. They are responsible for ensuring that materials 
are used as intended. This is to validate the research which is based on 
the experiments incorporated into the current operation of the course. 

Project roster . This has been a variable throughout the life of this 
project, particularly as to working personnel who generate products. At 
one stage during the life of the project the number of personnel was several 
times greater than the approximately 20 found to be associated with the project 
during the observation. For tnis reason, the roster can only hope to 
give a representation of staffing. 

1. Division Manager : The Division Manager is responsible for the company's 

interests at this location, and concerned with overall supervision of 
the project. This person was interviewed. 

2. Learning Materials Production and Development Manager ; The Learning 
Materials Production and Development Manager is responsible for 
coordinating all activities related to generation of the various 
classes of learning materials. The person holding this position 
had resigned short] y prior to the observation period and it appeared 
questionable whether this position would be filled for the remainder 
of the project duration. 



to Justify and somewhat meaningless relative to the real situation. 
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FIG . 3 . Project organizational structure . 
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Ld 1 t or ia I i) 1 rec to r ; lltiltorial director 

f!ilitorial contenL of instruct lotial naterial 
person was interviewed. 

W riting Sup ervisor: Tlie Writing Supervisor is responsible for 

allocation and coordination of activities undei the Editorial 
Director. ^ This person was interviewed. 

5. Audi o- Visual Specialist ; The Audii)-Vlsual Specialist is responsible 
To'F Reduction and quality of all itema within this category. This 
person was interviewed . 

Wr i ters /Analysts ; These people are responsible for 

producing textual material in its various forms. One was interviewed. 

7. Subject Matte r Kxperts ; These people work wit!) the writers to assist 
in generating appropriate material. In general, a writer and subject- 
matter expert engaged togetlier on the same textual material occupy 
facing desks. One was interviewed. 

8. Editors ; The f/litors are responsible for making copy ready for production 
typists . 

9. Production Typ ists; These people produce copy leady for reproduction 
by photo-offset or other means. 

10. Document Control ; Document Control has the responsibility of 
coordinating content and arrangement to conform with requirements 
of the research experiments. 

11. Contract and Business Administrator ; The Contract and business 
Administrator is responsible for contractual obligations in course 
production. This person was interviewed. 

12. Instructor ; The Instructor worked as an Academy Instructor preparatory 
to activity in the project. He now works as Instructor for the project 
and also within project management to provide needed interaction/ 
feedback. This oerson was interviewed. 

13. Course C jordlna tor (s) ; Course Coordinators are responsible, at 
respective locations, for coordinating course design and utilize- .ju/ 
Implementation. Tv^o were Interviewed. 

All of those personnel to the left of the chain-dotted line in 
Figure 3 work within the project, although one of the course 
coordinators maintains a post at the Academy. The Instructor also 
spends most of his time there. In addition to these people employed 
by the project, the following personnel identified with the Academy 
exert an Integral Influence on the project: 

lA . Chairman of Leadership Faculty : The Chairman of tho Leadership 

Faculty is responsible for integrity of course cont^ . to training 
needs of the Academy. 
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Director uf Programs : The Director of Programs is responsibio for 

coordination of objectives over all the experimental course projects. 
I'his person was informally interviewed. 



^ Gene ra ted 

Tin products generated within this project are subject to investigation over 
a varietiy of dimensions by which they may be referenced. Some idea of 
the sequence by whicli products get generated may bo discerned from the 
work flow ciiart presented aK. a generic time line indication in Figure 2. 

In terms of the focal products and clieir development, Figure 4 gives a 
schematic overview of the interrelationships. Figure 5 will give an 
evample of how individual products are organized within a typical part 
of the finril course. 

Index of production responsibilities . The following is an annotated 
list of the products identified in Figure 4, with arbitrary identification 
numbers cited for each. Those about which the observation team were able 
to obtain significant data, by suitable interrogation of people on the 
project whose tasks involved the generation of these products, are 
identified by the word ‘data.^* 

P-01, Acquisition of Original Course Segments from Academy . This 
was the material analyzed as a starting point for developing 
new material. 

P-02. Terminal Objectives for Course Segments . These were developed 
and revised for each segment v/ithin the course. (data) 

P-03. Enabling Objectives for Course Se^^ents . These were built 
from course content outlines. (data) 

P-04. Content Outlines for Course Segments , Involves the 

determination uf content and sequence within each segment. 

(data) 

P-05. Syndactic T^xt Modules for Course . These are multilevel 
texts built: in printed format around the content map. 

(data) 

P-06. Syndactic Text Modules with Remediation . These are identical 
to correspondingly numbered item within P-05, except that 
remediation is provided and they are relevant to the 
responses made by the student. P-05 does not provide this 
relevance. 

P-07. Single Concept Films . Mostly for use with depth core seg- 
ments (P-27), these elements form the basis for discussion 
exercise. (data) 
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P-08. Video Tape Modules . Used for presentation of lectures on 

video tape, these are in identical format with other modules 
of same segment for the core. (data) 

P-09. Audio Tape Modules . These are used for presentation of 

lectures on audio tape, in which the visual data (charts, 
pictures, etc.) presented on video are provided by panel 
books (P-10) . (data) 

P-10. Panel Books . These are books containing the visual s necessary 
to accompany lectures presented by audio tape. 

P-11. Learner Activity Summary Outline . A bibliographic session at the 
conclusion of a part of the course. 

P-12. Computer-aided Instruction (CAI) Modules . These are computer- 
programmed instructional modules to correspond with those 
using other media for the same segments* (data) 

P-13. Linear Programmed Modules . These are course content segments 
presented in linear programming mode, (data) 

P-14. Intrinsically Programmed Modules . These texts are designed on 
intrinsically programming principles, (data) 

P-15. Research Design Experiments . These were five experiments 

designed to obtain simultaneous data on selected variables. 

P-16 . Overall Research Design . A design and 

evaluation plan with the objective of providing information 
that will enable better utilization of respective media. 

(data) 

P-17. Instructional Presentation De-Jign . An arrangement for 
presenting the units of the course to optimize research 
and instruction value . (data) 

P-18. Major Parts of Core . These are 12 groups of segments on the 
major subject elements of the course. 

P-19. Criterion Referenced Test Pool . This includes approximo*:. iy 
1500 test items. 

P-20. Norm Referenced Test Pool . This includes approximately 600 
test items. 

P-21. Segments of Core . One or more (as many as six) modules (197 total) 
carrying identical content in different presentations for 
testing various parameters of presentation. 

P-22. Progress Checks . These are applied to the student after 
each module (or segment) with a view to determining his 
readiness for proceeding to the next segment (module) . (data) 
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P-23. Cumulative Post Tests . These tests are applied at the 
beginning of the course and after each part (13 in all) 
to assess student's learning at each major break in 
subject matter. 

Stude n t Data File . This provides a means of monitoring 
individual student progress and data each student providc's 
as research input. (data) 

^“25. Student/instructor Use Guide . A manual for use by students 
and instructors respectively as a guide for use of instruc- 
tional. material . (data) 

P-26. Course Developm e nt Model . The model for development based 
on original input with planned improvements and needs for 
instruction and research purposes. (data) 

P-27. Depth Core Segments . 'Tliese are various segments provided 
to deepen the impression and help the student apply the 
basic information learned in the core segments. 

P-28. Master Tutor Modules * Multitrack (4- track, in this instance) 
audio tape programmed to duplicate intrinsic programmed units 
on tape. (data) 

P-29. Structural Communication Modules . Another form of depth 
and/or enrichment unit, utilizing situation problem with 
matrixed response choice, aked to programmed instruction 
that enables the student to pvarsue his own course until he 
completely covers the available information within the unit, 
(data) 

P-30. CBI-70 Modules . An intrinsic programmed system, utilizing 
6-track sound film, presented on a CBI70 (Computer-based 
Instruction) machine. (data) 

P-31. Enrichment Units . These units utilize one of the types of 
modules in Products 28-30 to provide enrichment. 

P-32. Research Findings . The compiled results from the data 
collected in research. 



P-33. Multimedia Leadership Course . The ultimate text and lesson plans, 
etc. for continued use, after the experiment is completed, 
at the Academy. 

P-34. Reports for Various Agencies . These are technical and mile- 
stone reports (see Figure 2), including the terminal report. 



Content Map . This is an integral part, as well as an organi- 
zation tool, in producing each part of the course, showing 
the sequence and/or ordering of the segments within that part, 
(data) 





P-42. 
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Production responsibilities tree . The products within the production 
rerponslbilltles tree (Figure 4) are shown In clusters, providing visual 
evidence that numbers of modules go to make up segments, and segments make 
up parts. Figure 5 shows a typical part organization, from project documents, 
to Illustrate how the modules enable different students to be routed 
through different forms for research purposes. If this profile were 
reporting on the research of this project, each specific experiment would 
be described, together with the variables It addresses and how it addresses 
them. That is part of the purpose of the project being described in this 
profile. 

The form here is illustrative, showing the manner in which such an 
experiment can be organized. Each part is organized differently, for 
each purpose, so the precise arrangement shown in Figure 5 (shovm earlier) 
is only illustrative. 



Management Responsibilities 

The responsibilities identified in this section are those that 
emerged from interviewing personnel on the project with management respon- 
sibilities and from management activities in which those primarily occupied 
with generating products may also engage. As only some of the project staff 
were interviewed, the index listed here cannot claim to be complete. 

Production management , 

PM- 35, I nstructional Materials , This involves the management of 
production and delivery of the materials generated by the 
project, (data) 

PM-37, Contract (Proposal) . This involves monitoring the product 
development against the proposal specifications , (data) 

PM-40. Product Quality , This involves responsibility for all 
aspects of the product quality generated, (data) 

Environmental management . 

Budget , This involves reviewing performance against 
projected costs, in form and figures. (data) 

Staff Morale , This involves keeping everybody happy (if 
that is their best state) and willing to work. (data) 

Interagency Cooperation This involves seeing that the 
atmosphere between agencies involved is as good as possible 
for productivity , (data) 

Staff Hiring , Staffing the project with people who will 
work efficiently , (data) 







o 

ERIC 



EM- 36. 
EM-38. 
EM- 39. 

EM-41. 
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EM-43. Intraagency Cooperation . This involves keeping those 

witliin the project in the same state aimed at in EM-38. 

(data) 

EM-44. Project x\ccountabi 1 i ty . This involves seeing that the 
pxoject meets its comnilments . (data) 

>^nagement responsibilities network . Figure 6 shows a management 
responsibilities netv/ork linking together those responsibilities identified 
in the preceding index. 




FIG. 6. Management responsibilities network. 



Chapter III: Summary of the Data 



Data were gathered around the selected outputs by means of interviews 
with knowledgeable staff. The interviews sought to elicit for each out- 
put the standards by which the satisfactory completion of the output is 
judged, the tasks required to generate an output meeting those standards, 
and the enabler s (knowledge, skills, and sensitivities) which facilitate 
the carrying out oi those tasks. Interviewee statements were categorized 
subsequently into somewhat more general statements, for purpose of pro- 
viding more standardization of reported information. Tables 1-0 summarize 
the data in these categories by showing how frequently an item of Inter- 
view information was cited within one of these categories. 

Within each category are a series of descriptive labels which are 
representative of interviewee statements. These descriptive labels are 
listed in the tables under the category heading. In the process of 
reducing raw (interview) data, narrative interviewee statements about 
an output were linked to one of the category sets. Each narrative 
statement was then classified by means of a number code according to the 
most representative descriptive label within a given category or sub- 
category. 

Each table, therefore, provides the frequency with v^ich interviewees 
cited specific statements (which are represented by the descriptive labels 
in the tables) of standards, tasks and enablers in relation to each 
output . ^ 



If the reader is interested in the narrative statements of the interviewee, 
these can be found in the Appendix. To locate the narrative statement for 
any given category, first note the output and its identification number in 
the table. Second, note that each descriptive label within a given category 
has a distinct number or code. Turn to the Appendix and locate the output. 
Under the output locate the category label or heading (standard, task, or 
enabler) and pinpoint the number or numbers (depending on the frequency 
cited) of the descriptive label which appeared in the table. The statement 
in the Appendix opposite this number is the original narrative statement 
from an interviewee and is only represented in the tables by its descriptive 
category label and code number. 
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P-26 Course Development Model 

P-28 Master Tutor 

P-29 Structural Coaaiunlcatlon 

PK-35 Instructional Materials 

EM-36 Budget 

FM-40 Product Quality 
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Chapter IV: Supplementary Data 



This chapter contains information about staff background, the re- 
sources and equipment used in carrying out the tas o of the project, and 
the classif ications of output characteristics. 

Summary of Staff Background 

Of the eight personnel who completed the Job/Task Inventory forms 
as part of their interview sessions, one had a doctorate degree, three 
had master's degrees, and four had bachelor's degrees. Their specialties 
were all different, and were as follows: 

Education: Teaching and research 

Education : Teaching and guidance 

Personnel administration 

German language 

Engineering 

Economics 

Business administration 
Broadcasting and public administration 

The experience represented in these personnel aggregate 16 years 
teaching in college or university settings, five years conducting research 
in similar settings, one year teaching in public school, five years working 
for state or national educational agencies, 11 years working in educational 
R & D centers, 12 years working in the present organization (from 1 to 
4.5 years each), and eight years in other educational or research employment. 

In response to questions on training that personnel had, or thought 
necessary for the work, each named training specific to che job (see 
project roster in Chapter III), plus emphasis on the part of several 
writers on experience in military training background. 



Support Resources 

Those support resources used by the personnel interviewed and seon 
as necessary included: equipment construction (mechanical, electronic, 

carpentry, etc.); printing; slide, film, and tape reproduction services; 
photography; artwork and illuscrations; drafting; technical writing; 
editing; secretarial service, other than typing; typing; purchase of 
supplies and equipment; library holdings; subscriptions to technical and 
professional journals/periodicals; requests for documents or publications 
not locally available; computer analysis services (data processing); 
computer program ’/riting; statistical consultation; audio- visual aids 
and devices; subjects for experimentation or try-out of procedures; 
travel arrangements; budgetary and other fiscal accounting; scoring 
of test items; television facilities and equipment. 
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Support Eqiipment 



Listed as available to individual personnel were: dictation equip- 

ment, desk calculators; photographic equipment; video tape; television 
camera; readers for microfiche or microfilm; and audio tape recorder. 



Classifications of Qu tput Characteris tics 



As the Oregon Studies evolved it became evident that outputs could be 
categorized in terms of a number of variables. Among them are (^) Structure 
(product, event, or condition), (^) Function (policy setting, management, 
or production), {c) Level (focal, component, or facilitating), (<d) Charac- 
ter (knowledge, technology, implementation, or information), and (je) Stage 
of Completion. Tliese five schema are represented in Table 7 for each 
project output id.*intified, with frequencies summarized for each category* 
Table 7 h \s been added to this profile subsequent to the profile's original 
writing. 
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TABLE 7 

Clflttlflcflt loot of Output CharacterltClct 





Output Characteristic" 


Project Outputs 








Charac ter 






Structure 


Function 


Level 


(Products onlv) 


Completion Stn^e 


No. Ubel 


P e c 


PS tl p 


fl c fj 


k t ll 1? 


12 3 4 5 6 


P-Ol Acquisition of Original Course 


X 


X 


X 




X 


Stsments froa Aesdeay 












*P'02 Ternlnel Objectives for Course Segments 


X 


X 


X 


X 


X 


*P'03 EnebLlng Objectlven for Course Segmtote 


X 


X 


X 


X 


X 


*P~04 Content Outlines for Course Segments 


X 


X 


X 


X 


X 


♦P-05 Syndectlc Text Modules for Course 


X 


X 


X 


A 


X 


P-06 Sjmdertlc Text Modules with Remedletlon 


X 


X 


X 


X 


X 


*P-07 Single Concept Films 


X 


X 


X 


X 


X 


*P-08 Video Tepe Modules 


X 


X 


X 


X 


X 


*P-09 Audio Tepe Modules 


X 


X 


X 


X 


X 


P-10 Penel Books 


X 


X 


X 


X 


X 


P-11 Leerner Activity Sussaery Outline 


X 


X 


X 


X 


X 


*P-12 Computer-elded Inetruction Modules 


X 


X 


X 


X 


X 


•P-13 Linesr Progrened Modules 


X 


X 


X 


X 


X 


•P-14 Intrineicelly Progremaed Modules 


X 


X 


X 


X 


X 


P'lS Reeesrch Design Experiments 


X 


X 


X 




X 


•P-16 Overell Reeesrch Design 


X 


X 


X 


X 


X 


•P-17 Instruct ionsl Preeentetion Design 


X 


X 


X 


X 


X 


P-16 Hejor Perts of Core 


X 


X 


X 


X 


X 


P-19 Criterion Referenced Test Pool 


X 


X 


X 


X 


X 


P-20 Norm Referenced Test Pool 


X 


X 


X 


X 


X 


P-21 Segments of Core 


X 


X 


X 


X 


X 


•P-22 Progress Checks 


X 


X 


X 


X 


X 


P-23 Cumuletive Poetteete 


X 


X 


X 


X 


X 


•P-24 Student Date File 


X 


X 


X 


X 


X 


•P-25 Studenc/lnetructor Use Guide 


X 


X 


X 


X 


X 


•P-26 Course Developsent Model 


X 


X 


X 


X 


X 
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Claialf Icatloni of Output Charactertitlci 





Output Characteristic® 


Project Outputs 








Charac ter 






Structure 


Function 


Level 


(Products only) 


Completion Stane 


No. L.nbcl 


P_ • c 


ps m p 


f 1 C f 2 


k t li 12 


1 2 3 4 5 6 


P-27 Depth Core Segments 


Z 


X 




X 


X 


*P-2N Master Tutoi Modules 


Z 


X 


% 


z 


X 


AP-29 Structural Cooanunlcatlon Modules 


z 


z 


X 


z 


X 


•P-30 CBI - 70 Modules 


z 


z 


X 


X 


X 


P-31 Enrichment Units 


z 


X 


z 


X 


X 


P-32 Research Findings 


z 


z 


X 


X 


X 


P-33 Multimedia Leadership Course 


X 


X 


X 


X 


X 


P-34 Reports for Various Agencies 


X 


z 


X 


X 


X 


PH-35 Instructional Materials 


z 


X 


z 


X 


X 


EM-36 Budget 


z 


z 


z 


X 


X 


PM-37 Contract (Proposal) 


z 


X 


X 


X 


X 


EM-38 Staff Morale 


z 


z 


X 




X 


Dl-39 Interagency Cooperation 


z 


z 


X 




X 


PM-40 Product Quality 


z 


z 


X 




X 


EM-41 Staff Hiring 


z 


z 


X 




X 


•P-42 Content Hap 


z 


X 


X 


X 


X 


Dl-43 Intra-agency Cooperation 


X 


X 


X 




X 


EM-44 Project Accountabllltv 


X 


X 


X 




X 


Classification Frequencies^ 


36 3 5 


0 17 27 


3 23 16 


1 31 1 3 


7 10 2 17 1 7 



■ The specific output characterletlce are Identified aa followe; 



Structure 



runctlon 



Lervel 



Character 



CoapXetloQ Staae 



p - product 
a - event 
c - condition 



pa - policy aettlng 
■ - aumageoent 
p - production 



fl - focal 
c - coaponent 
f 2 - facilitating 



k 

t 

11 

12 



knowledge 

technology 

lapleoentatlon 

Information 



*^ata totals In this table may vary allghtly from data In tables reported elsewhere. This la 
governing clasalf Icatlon of outputs having been revised and applied to these data subsequent 
proflia. 



- completed over one year ago 
~ completed 3 to 12 months ago 

- completed within last 3 mos. 

- currently in progress 
" not yet underway 

on going (continuous) 
a function of decision rules 
to the preparation of the 
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Chapter V: Project Dynamics 



On Primary Focus 



The 8omewh:^t unusual feature about this project is its dual foci: i.e., 
research and development. In some senses, virtually everything done within 
the project is Influenced by both foci: the material produced must be good 

for both purposes, the course being developed for the Academy, and the 
research being for the benefit of the agency conducting the project and 
the funding sponsor. 

As tends to happen in aich a situation, because different personnel 
are assigned to responsibilities connected with one focus or the other, 
individuals are inclined to favor one or the other viewpoint where 
differences arise. 

It would seem that, what is good for one is good for the other if 
good research results in a contribution to knowledge about learning that 
in turn results in the development of a better course. This concept 
postulates a relationship that has been presented often for the world of 
R and D. It does not always seem to work out that way in the real world. 
Things that happen within this project well illustrate this departure. 

One “ideal" for a project such as this would be to start with course 
material as it presently exists, make definitive changes, predominantly 
related to the utilization of various media (since this is to be a multi- 
media course) then measure the effect of such changes according to reliable 
research principles. This should in turn result in decisions regarding the 
best format, content, media presentation, etc. at each point along the 
curriculum in the course. Theoretically the course will progressively 
improve if this process is followed. 

This is not always what actually happens. To get a better picture of 
what work in such a situation is like, w* need to examine the causes for 
departure from the foregoing “ideal." 



Nature of the Research 



While a profile such as this cannot describe the research design 
in detail, an understanding of the nature and postulates of the research 
is valuable tc understand how conflict with course development can generate. 

The research cited here is concerned with the proposition that media 
have individual characteristics of their own that can influence learning 
and that the learning which results from media characteristics is independent 
of the learning of “content" which results from media use. Thus, from the 
research viewpoint, the course design is not intended to optimize content in 
each medium so as to best utilize that medium relative tc the specific sub- 
ject matter, but rather to determine possible comparative limitations asso- 
ciated with different media for making identical content presentations. 




i 
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Most of the units (modules) connected with the research employ 
content designated as verbal and visual — a composite of the two. Pictorial 
presentations are made, and words are presented before, after, or with the 
presentations. By regarding the total presentation in a specific medium as 
a “system,” the research postulates capability variables after Tosti and 
Ball,^ which are set forth in the matrix and dimensions shown in Table 8. 

Three capabilities are distinguished: stimulus, response, and manage- 

ment. Each may vary in form and in frequency or timing. 

Stimulus refers to the means by which communication of content reaches 
the student. This can be ver’cal-written , usually a printed page; verbal- 
spoken, such as a teacher talking; and pictorial, such as something drawn 
on the chalk board, projected by an overhead projector, or presented as art 
work in the textbook. Briefly, this divides stimulus into words and pictures, 
with the word presentation being subdivided according to whether the words 
are visually or aurally perceived. 

Duration of stimulus is an interesting factor. When the student reads 
the printed page and looks at the pictures, he can regulate his own speed. 

The medium itself is persistent. When the student is presented a sound 
film or a video tape, both words and pictures are transient: he has no 

control over how long he can look or listen. 

An audio tape with a panel book splits the stimulus on the frequency/ 
timing dimension: the words are transient, but the pictorial contained 

in the panel book is persistent. Relative to form of stimulus, much has 
gone before relative to whether the verbal should be visual, aural, or 
both . 



Stimulus relates to communication one way: to the student. Response 

is where the student is expected to communicate back. Three forms are 
distinguished: overt, which may be written or spoken, with well-known 

advantages and disadvantages when students work in a group; and covert, 
where only the student knows his response: it is not communicated to the 

teacher. 

The effectiveness of presentation upon learning is postulated as 
varying, possibly in different ways with varying content, according to 
the frequency df response demand, as well as the form. How often is 
the student asked a question relating to vhat he has heard or read; once 
a minute, once every three minutes, etc., or not at all? 

The final capability within the matrix is management, a term which 
refers to a basis for a decision to change the presentation, based on 
student response. \^ere the response is overt, the basis for decision 
is obvious. Where it is covert, possibly the expression on the student's 
face provides a clue, or the student may only make a personal note for 
his own future reference. 



2 

“A Behavioral Approach to Instructional Design and Media Selection" 
by Tosti and Ball, 1969. 
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TABLE 8 

System Capability Matrix and Dimensions of 
Presentation 



System 


Attributes of System Capability 


Capability 


Form 


Frequency /Timing 


Stimulus 


Stimulus Representation 


Duration 




Verbal - written 


Transient - persis tent 




Verbal - spoken 


Length of time the 
presentation remains 




Pictorial 


intact : 

(^) low 

(b) intermediate 

(c) high 


Response 


Response Demand 


Respons e-Demand Frequency 


Overt - written 


Infrequent - Frequent 




Overt - spoken 


Frequency of response 
required ; 




Covert 


(a) low or zero 
(h) intermediate 
(c) high 


Management 


Management Form 


Management Frequency 




Repetition 


Infrequent - Frequent 




Multilevel 


Frequency of decision 
to change presentation: 




Multiform 


(a) low or zer* 

(b) intermediate 




Error-diagnostic 


(c) high 
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Four forms of management selection are considered: repetition, 

wliich means the part suspected of not being absorbed by the student is 
merely repeated, in whole or in part; multilevel, which means the remedial 
material provides material at a different level; multiform, which means 
that the form of presentation is changed, probably for reinforcement; and 
error-diagnostic, which means the student’s incorrect response is 
correlated with a basis for error, and remediation is administered based 
on that diagnosis. 

Management frequency can vary, as can response demand frequency, on 
which it depends. Management frequency may be the same as or lower than 
response demand frequency. 

The research on this project aims at measuring along these different 
capability dimensions, based on routine, through different modules 
at different stages through the course. 



Practical Li mitati ons to the Rese arch Ideal 

An essential feature of such research is that the system capability 
be changed only in the dimensions tabulated. This sounds simple to do. 
At one stage, it was argued that to achieve this the spoken word used 
for comparison must be equally emphasis free, because a student reads 
the printed page without emphasis. The emphasis he must supply for 
himself . 

As of the time of observation, video tapes were being made with 
insistence that the verbal presentation content is word for word the 
same as the texts against which it is compared, although readers were 
sought who either knew the subject, or, by being professionals at this 
work, sounded as if they did! 



Programming Li m itations 

Whether programmed modules used a printed text or a taped or filmed 
counterpart, the research design resulted in an insistence on identical 
presentation. This presented difficulties of a purely mechanical nature. 
One of the writers of intrinsically programmed texts Insists that the 
programming results in the student who takes a ’’correct” course doing a > 
minimum of page turning, while any reference in the programming (resulting 
either from correct or incorrect responses) results in the student 
turning not more than eight pages to find the next frame. 

This establishes a standard for ease of use with printed programmed 
texts. Film and tape versions of programmed instruction have no corre- 
sponding dimension, because automation of program frame selection, keyed 
to student responses, takes ’’frame finding” out of the student’s hands. 

Another basis for difference involved the need for using a reader 
who could suitably inflect his voice for the tape recorded modules (audio 
and video) and rewriting the printed text so a reader was less likely to 
encounter ambiguity. But oral expression can eliminate ambiguity in the 
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spoken word, so this raises the problem: do both texts have to be 

changed to maintain identity? Research design insists but 

magnitude of effort Imposes constraints. 



Related Conflicts 

These basic conflicts between research and development have shaped 
differences of opinion ever since the project began. But this has not 
prevented tremendous strides from being made. To appreciate what it is 
like to work on this project one must see how this central conflict, 
which evidences itself in a variety of ways, creates both conflicts 
between goals and conflicts between people. 

The whole design of the project has become exceedingly complex, 
as the production responsibility tree (Figure 4) shows. And that is 
only one form in which such a project tree could be drawn. The tree 
would become quite complex if the various steps, shown in the tree as 
clusters of subproducts, were delineated. 

As this profile states, the dual foci of the project produce products 
of Interest to different agencies: the Academy gets the course developed 

and WLC gets the benefit of the research for future projects. This 
naturally tends to mean that the Academy staff emphasizes the need for 
a quality course and tends to object if or when the needs of research 
appear to be intruding against that objective. 

The previous statement could be misleading without qualification. 

The conflicts evident present some apparent contradictions. According 
to policy both the project staff and the Academy staff are committed to 
the need for change. Evidence from the project suggests that the need for 
change, rewriting, and modifying, versus the motivation to retain material 
as it is, finds mixed supporters in each agency and according to individual 
c Ir cums tances . 

A principle that is unquestioned, but that periodically creates a 
stir, is that the language of the course shall be ”Navy'* where references 
to such things as chain of command occur. 

As well as the obvious conflict between the goals of research and 
development, with some more or less general alignment between resea^c' 
and the project staff, and between development and the Academy staff, 
another goal Intrudes Itself into the picture: that of the midshipmen. 

Their goal is to get satisfactory grades in their course work. Their 
main concern Is with graduating on schedule. 

The Leadership, Psychology and Management course is just one course 
of many to these students, albeit an Important one. Although they are 
subjected to talks and "advice" with the purpose of encouraging them to 
cooperate with research aspects of the course development, their primary 
objective as individuals is, understandably, to secure high grades. 
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Kor example, one objective of the research is to compare the use of 
presentations that employ transient stimuli for both verbal and pictorial 
components, with the use of presentations that employ transient verbal 
stimuli with persistent pictorial. This involves video tape for the 
former, and audio tape witli panel books for the latter. 

Due to pressure on their time, each student is concerned, as far as 
possible, with selecting the media which suit his individual needs best. 

Thus, one student will prefer audio tape and panel book. One such student 
commented that he could listen to the tape while doing something else, 
such as shining his shoes, and study the panel book at his leisure. 

Another student may prefer the video tape, because he has to concen- 
trate during the transient presentation and finds himself able to retain 
sufficient information that way to obtain adequate grades, without rein- 
forcement by a more persistent presentation. Video tape thus optimizes 
this student’s utilization of his time. 

To average out differences due to individual student learning 
characteristics the intent of the research design is to route students 
so individuals get different media during different segments of the course. 
Different students are interchanged in varying sequence, to enable the 
effects of different content to be separated from effects due to difference 
in student preference , etc . 

However , the fact that each student’s individual goal is to attain 
satisfactory grades causes the individual student to use ingenuity to 
deviate from his assignments. What matters to the student is achieving 
qualifying grades at each segment of the course, regardless of which 
module is studied (e.g., audio or video tape) during any particular seg- 
ment preparatory for the test (which is the same whichever presentation 
module is used for that segment). 

Thus, a student who prefers audio tape may borrow this presentation 
from a classmate who has that medium assigned to him, without notifying 
the faculty or coordinating staff. Unless this deviation is discovered 
it can serve to invalidate the data emerging from the research, because 
results actually obtained (in this instance) by studying with audio tape 
get credited to use of the video tape module. 

Various measures are used to track this kind of deviation, including 
one that was somewhat accidental. One of the coordinating staff has apparent 
capacity for establishing very good rapport with the students . She does 
her best to persuade them to comply and, whether or not she succeeds every 
time, she keeps quite close tabs on them. This helps to keep the records straight. 

Character of the Course 

The character of the course can be described by comparing its 
subject matter with that of the courses that formed the subject of two 
other projects, not the subject of this observation. One of these was 
a science course, the other an economics course. Science is viewed as 
a "hard” subject, dealing with well-established spe ifics. Economics 
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may not be as hard as science, but neither is it as ’’soft** a subject 
as Leadership, Psychology and Management, 

This last course deals exclusively in relationships with people, 
and thus is the ’’softest’* course of all three. This compounds the problem some- 
what, Thegeneral subject area overall is one in which authorities differ 
more widely than they do in either of the other subject areas, science 
or economics , This is no t really one subj ect area , but three overlapping 
areas, leadership, psychology, and management, in which the best authori- 
ties in one area differ with the best authorities in another, 

A considerable proportion of the studies in each of these subject 
areas has taken place in other than services situations. The present 
course takes cognizance of this by presenting a broad coverage of 
different concepts of, or bases for, leadership , with the emphasis being 
placed on the variety of roles the Academy graduates will have to exercise. 

Even with such prescriptions, the qualities of leadership are 
somewhat intangible, dependent on individual personality traits that 
each can ultimately utilize in a manner peculiar to himself. Leadership 
or management "style” may not vary as widely in a service setting as it 
does in a business or industrial setting, for example, but in the last 
resort, it is still very much subject to individual idiosyncracies , 

How do these variables influence a multimedia course, the development 
of which is already combined with a research design? They do not simplify 
matters! It is to the credit of the project that the Director of Programs 
at the Academy, who oversees all three projects, while commending each 
project for good features in the way that project was handled, commended 
this one on the professionalism with which it was handled. 

To interpret such a commendation requires an identification of 
precisely what he meant by "professionalism,” But that is another 
intangible. This was his impression, as a professional himself. His own 
presentation on the matter suggested that his area of expertise is in 
instructional design: he is intensely aware of the complexity of the 

dimensions involved in the programs he is directing. 




Chapter VI: Training Implications 



On this project, every man hired and every subcontractor engaged was 
selected on the basis of competence for the particular item or area of 
work involved. For this reason, any attempt to list training suggestions 
obtained from this project would read like a duplication of many of the 
enablers listed under the products connected with the work involved. 

A strongly emphasized common demoninator , across many of the staff, 
was the inclusion of a background from the Naval Academy, or from some 
other service academy, possibly with subsequent Nav>' experience. While 
nor every individual engaged on the project had such a background, by 
far the majority of the responsible personnel did. 

As people were used for the documented backgrounds and the demonstrated 
competence they possess, and some filled more than one ''slot" on this basis, 
it would be difficult to enumerate training implications, either for 
general work on such a project, or for work on specific parts of it. 

Rather, if a person possessed some expertise that fitted a "vacancy" within 
the project, then he would be eligible to fill that spot. 
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^ Appendix A: Liating of Product Standards, Tasks, and Enablers 
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The following listings are headed by the products identified in the 
previous chapters. As the listing is rather extensive — although not 
exhaustive by any means — no comments are offered, unless there appears 
to be some specific reason for comment. These statements were extracted 
from discussions with interviewees and were coded into their respective 
category sets. The selected code precedes the statement and Indicates 
the following for: 

STANDARDS 



Code J: Structure of Standards. 

J-1 Standards against which outputs are judged, (output 
oriented) 

J-2 Standards against which processes and/or operations are 
judged. (process oriented) 



Code LM: Primary Categories of Standards. 



TASKS 

Code NO: Clusters of Tasks 

ENABLERS 



Code S: Structure of Enablers. 

S-1 Knowledge. 

S-2 Skill or ability to perform. 

S-3 Sensitivity or awareness. 

Code UV : Primary Categories of Enablers (knowledges, skills, or 

sensitivities) . 



The codes associated with these three categories (standards, tasks, 
enablers) are the same both here in the listing and as previously cited 
in Chapiter III tables. 



P-02: Terminal Objectives for Course Segments 




STANDARDS: 



J 

T 

1 



ML 

04 

19 



Terminal and enabling objectl\^9 are clear /understandable. 
Objectives are r'^lated to tiii^kufcerlals . 
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1 09 Modified ohjec wives agree with changes in the text-content 
outline . 

1 14 Acceptancy by sponsor. 

TASKS : 

W 

01 Analyze former course for areas of learning It covers, results 
It expects. 

02 Write objectives for each section working from new mediated 
course content. 

02 Negotiate eacli objective with user for acceptance. 

04 Build one terminal objective per major subdivision of every 
content outline. 

06 Revise objectives on basis of added/deleted materials in content 
outline , 

06 Delete objectives on basis of revised content oullin-. 



ENABLERS : 

1 08 Knowledge of course content in terms of what/where material 
covered . 

1 08 Knowledge of what user wants taught. 

1 03 Knowledge of purpose of terminal objectives, enabling objectives. 

2 14 Skill in writing objectives, 

2 02 Skill at negotiating acceptance or modification of each 
terminal objective with user. 

2 02 Ability to use diplomacy in working with writers. 

2 06 Ability to organize skill in explicating objectives . 

2 18 Ability to correlate material with terminal objective. 

2 08 Ability to read carefully — to see that the objectives and 

material correlate and correspond . 

3 10 Sensitivity to the interrelationships in deriving enabling 

objectives from terminal objectives. 



P-03: Enabling Objectives for Course Segments 



STANDARDS : 

j m 

1 14 Acceptance by sponsor. 

TASKS : 

04 Build enabling objectives from course content outline. 

21 Work with writers on developing enabling objectives for each 
terminal objective. 
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(Notice different approach adopted in these tasks,) 



ENABLERS : 
S UV 



1 08 Knowledge of what sponsor waiits, 

1 03 Knowledge of purpose of terminal objectives, enabling objectives. 

2 14 Skill in writing objectives. 

2 02 Skill at negotiating acceptance or modification of objectives 
for each section with sponsor. 

2 02 Skill in diplomacy in working with writers. 

2 06 Skill in organizing/explicating objectives. 

3 10 Sensitivity to interrelationships in deriving enabling objectives 

from terminal objectives. 



1 14 Acceptance by sponsor. 

1 09 No parts missing that are important. 

1 07 All parts are clear. 

1 09 Material is not redundant. 

1 09 Material is not oversimplified. 

1 04 Clarity — material is clear. 

1 04 Conciseness — material is ‘:oncise, 

1 17 Examples are appropriate. 

1 19 Examples are ”good** for point they are to make. 

1 07 Examples are not too dramatic. 

1 05 Material is appropriate for the target population — the 
content of the material is appropriate. 

1 05 There is no unnecessary information (too much detail/depth) in 

content outline. 

2 07 Writers have completed the task or product. 

2 04 Schedule adequate — deadlines have been met. 

2 08 Writers' questions have been answered. 



NO 

02 Work with context analysis and writers to clarify course content. 

03 Determine final format of content outline. 

03 Relate all objectives to former course areas. 

03 Decide on new weighting of parts — which areas should be expanded, 
which contracted. 

04 Build content outline by topic which will work in the whole, 
part traditional, earlier coarse. 




STANDARDS : 
J LM 



TASKS : 



ERIC 
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04 Assemble parts of content outline into booklet form. 

04 Type content outline in standard format. 

06 Remove redundant material from content outline. 

06 Rearrange material within content outline. 

06 Revise content outline based on comments from implementations 
supervisor and subj ective opinion . 

06 Submit content outline to review for recommended changes. 

06 Correlate/correspond content outline with course materials. 

06 Insure same verbiage between content outline and materials. 

06 Add/delete examples where necessary. 

06 Determine parts that must be retyped or where can cut and paste. 

06 Review for mechanical errors. 

06 Revise content outline based on current needs and changes. 

06 Study outline and suggest deletions and changes. 

06 Delete nonessential material from content outline. 

06 Compare subcontractor’s recommendations against original list 
of items on contract outline. 

06 Check to see if course outline includes all parts I feel necessary 

07 Negotiate with sponsor for acceptance of each part. (This has 
never stopped-still going on.) 

07 Work toward concurrance/nonconcurrance on recommended changes. 

12 Assign writers’ tasks — corresponding/correlating content outline 
with course materials — based upon revisions. 

12 Determine amount of time it would take a writer to review 
correspond/correlate content outline with course materials. 

12 Deteimine a production schedule. 

12 Interact with consultant about writing outline. 

14 Submit content outline and recommended changes or modifications 
to implementation supervisor for review/ app roval . 

14 Provide writers with clarification in relation to corresponding 
content outline and course material. 

14 Double check writers* work against revision specifications to 
see that specified changes were made as required. 

14 Apply for criteria; clear, concise, complete, and correct 
to revision process. 

14 Ask other members of the staff for their judgment on technical 
topics . 

14 Ask sponsor to resolve conflict over quality issues. 

19 Negotiate with sponsor for emphasis, agreement on weighting of 
subj ects of course . 

20 Discuss in conferences the performance of staff, i.e., the writers 



ENABLERS : 

s uv 



23 

05 

05 

04 

05 
08 
05 
08 



Knowledge of what midshipmen already know. 

Knowledge of military circumstances through experience. 
Knowledge through experience of midshipmen's life. 

Knowledge of programmed instruction , principles and techniques , 
Awareness of sponsor's standards and context. 

Knowledge of traditional previous course content. 

Knowledge of context, methods and experiences. 

Knowledge of mechanisms/operations (within content) to make 
recommendations on the content outline. 






1 07 Knowledge of multimedia/mode used with a particular segment 

(knowledge of what syndactic, linear tests, etc. are). 

2 06 Ability to write programmed frames of instruction. 

2 32 Ability to proof read for mechanical error. 

2 18 Ability to proof read for consistency. 

2 11 Thoroughness in work — ability to be thorough. 

2 05 Ability to plan work — in terms of time and space. 

2 14 Skill in writing clarity, etc. 

2 02 Ability to work with sponsor personnel in negotiating content 
of course outline, changes, etc. 

2 11 Ability to persevere in determining sponsor’s desires thru 
negotiating. 

2 18 Skill in determining what is nonessential in content outline. 

3 05 Previous experience with content materials— helpful . 

3 30 Sensitivity/awareness of end-use of the product — used at academy 
by second-class midshipmen. 

3 30 Sensitivity to the context (user agency) — related to the above 
sensitivity . 

3 31 Awareness of the effect of changes on the rest of the materials — 
awareness of the interrelationships of the materials. 

3 02 Appreciation for time and what can be done in a time segment. 

3 17 Awareness of personalities of staff you are working with. 

3 02 Awareness of staff capabilities in relation to the job. 

3 31 Awareness of the nature of the product and the steps/pieces 
involved in producing that product. 



P-05: Syndactic Text Modules for Course 



STANDARDS : 

J 

1 07 It looked all right to me. 

1 07 Concise — my feelings . 

1 01 Self-check that product is complete (covering of all obje'^t ^es) , 
correct . 

1 01 Material correlated with content outline and the terminal and 
enabling objectives. 

1 04 Language is appropriate, i.e., material has been stated in 
best possible way. 

1 12 The final statement reflected the objectives clearly. 

1 15 There is no redundancy, it is concise enough. 

1 08 Clear — concensus of staff. 

1 01 Complete — reference is to content outline. 

1 24 Correct — military jargon, etc. 

1 13 I accept opinion of other staff more expert in content areas. 

1 13 Chief editor accepts the unit. 
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802 



1 13 Approval by subject matter expert of any examples of military. 

1 14 Acceptance by sponsor. 

1 13 Editor approves clarity, corrections and grammar, spelling. 

1 06 Students found material interesting. 

1 04 Test data does not show any frames to be misleading/ confusing . 

TASKS : 

W 

01 List the topics to be covered in text unit. 

01 Read material primarily texts in the office — in relation to the 

subject matter of a particular syndactic text (literature review). 

01 Collect textbook material to read/review relative to content. 

01 Consider level of material in terms of audience (midshipmen). 

02 Determine which material (from textbook review) will be used to 
support a particular point relative to subject matter. 

03 Specify and/or create graphic materials to be included in 
syndactic texts. 

03 Number the topics in sequence of presentation. 

03 Correlate content outline with material for syndactic text. 

03 Determine number of summaries to be included within each syndactic 
text-— there, are major parts/sections of the text. 

03 Establish guidelines/steps to follow in writing a syndactic 
text format . 

04 Use progress checks as a reference in writing materials. 

04 Think of original/sensible ways of presenting and testing 

information/content . 

04 Write test items to cover basic points in the syndactic text. 

04 Write summary/overview part of the text. 

04 Write frames — an expansion and check of points covered in 
original summary . 

04 Scramble/rearrange posttest items and answers — same as pretest 
items after summary. 

04 Obtain the master material to be summarized. 

04 Obtain the terminal objective for this unit. 

04 Rewrite the topic titles to fit into space required. 

04 Write new summary of segment which needs revising, accommodating 
criticisms of information . 

04 Write new linear frames to go with summary. 

04 Write new summary quiz to match new summary in content. 

04 Produce the interview form for interviewing midshipmen. 

04 Type material for camera copy readiness. 

05 Interview midshipmen about effect of product. 

05 Use an interview form during the interview of the midshipmen. 

06 Utilize revision related comments from the user agency as to 
major points to cover in syndactic text . 

06 Modify content outline on basis of readings (literature review 
relative to syndactic text) . 

06 Review syndactic text in relation to content outline to make 
sure every point in the outline has been treated/acknowledged, 

06 Check to see that progress checks correspond until material is 
actually taught . 
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06 Check that terminal objectives and enabling objectives for the 
segment were covered. 

06 Edit entire text of segment for more types. 

06 Improve phraseology throughout entire segment of course. 

06 Read the segment which has been sent back for revision as it 
stands, and read comments which indicated need for revision. 

06 Check product against criteria, 

06 Prepare recommendations for modification. 

06 Make recommendations for changes in instructional material. 

06 Proof read material — typists/production/writer/editor. 

06 Modify/incorporate recommended changes from user agency and editor. 

12 Discuss instructional examples with subject matter expert for 

correspondence with reality of military experience. 

14 Send over pieces of the text for the user agency to review and 
make suggestions and/or corrections . 

14 Review materials sent by the contractor to the user. 

16 Assist other writers in writing a syndactic text. 

19 Interact with sponsor as to nature of the material in syndactic 
text . 

21 Provide all writers with a set of format guidelines. 

21 Hold committee meeting to review depth core materials. (Those 
so specific it required more specific expertise.) 



ENABLERS : 
S W 
1 08 

1 03 

1 02 
1 08 
1 07 
1 19 
1 06 
1 12 
1 17 
1 08 
1 08 
1 17 
1 19 
1 08 

1 04 

2 14 
2 06 

2 10 

2 19 

2 33 
2 27 



Knowledge of content outline so that proper objectives are 
covered in revised form. 

Knowledge of function of TO’s and EO*s (terminal objective- 
enabling objective). 

Knowledge of syndactic course structure. 

Knowledge of leadership course content. 

Knowledge of the media being used for each section. 

Knowledge of jargon used by midshipmen. 

Knowledge of military practices and terms used. 

Knowledge of who to ask if knowledge not in head. 

Ability to write concisely. 

Knowledge of objectives in course and specific unit. 

Knowledge of contents of course and specific unit. 

Knowledge of rudiments of writing (instructional materials). 
Knowledge of Navy jargon. 

Superficial/introductory knowledge of content area of syndactic 
text . 

Knowledge of how to write test items — primarily multiple choice 
type Items. 

Ability to write a paragraph — related to the summary. 

Skill in ways of presenting and testing material — frame writing 
styles . 

Ability to work with data — read data sheets to pin point test 
items that might need revision. 

Ability to specify graphic illustrations/charts to accompany 
text . 

Ability to draw the illustrations/ figures to accompany text. 
Ability to type. 
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2 26 Ability to look at index and determine where most useful 

information relative to a particular topic would be found in 
the publication. 

2 06 Ability to maintain a continuity or flow in the sequence of 
frames that are written in a syndactic text. 

2 09 Ability to logically approach a subject or topic. 

2 14 Skill in writing. 

2 11 Ability to be persistent and painstaking in reviewing materials. 
2 01 Experience in teaching subject matter. 

2 1^ Ability in writing in order to write readable examples, 

portions of text. 

3 17 Sensitivity to the specific characteristics of target students. 

3 20 Sensitivity to needs of students beyond class circumstances. 

3 27 Sensitivity in the balance in writing a paragraph — points in 
paragraph as specified by topic sentence have been covered. 



P-07: Single Concept Films 



STANDARDS : 

j m 

1 06 Product holds interest of students. 

1 12 Communicates as registered by statistical results. 

1 14 Approval of sponsor. 

1 16 Does it measure up to user's expectations, e.g., commercial TV. 



TASKS: 

m 

03 Prepare scenario for film presentation to show context of 
'’incident.** 

03 Work on shots to effectively present problem or question. 

04 Write a script of incident for depth core module. 

05 Pose problem for class discussion. 

12 Organize filming schedule — about one day for a 3-4 minute film. 

13 Purchase 16mm camera, plus necessary accessories. 

13 Obtain acting talent from sponsor. 

14 Have script approved by sponsor. 

04 Rehearse performances, coach them. 

06 Edit film (sit down with professional film editor). 

14 Submit to viewing by academy for review. 

ENABLERS : 

S W 

f 22 Knowledge of A-V hardware for media. 

1 04 Knowledge of production technique relevant to individual media. 
1 17 Knowledge of audio and print writing techniques. 
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2 15 Ability to read as if knows material (narrator). 

2 16 Ability to adapt script, etc. to each medium. 

2 18 Ability to fit everything into scope of medium being used. 
2 20 Ability to make creative decisions. 

2 14 Ability to write for both audio and print. 

2 14 Ability to write dialog — not stilted as for print. 

3 27 Sensitivity for taste in varying shots. 

3 13 Use of concise, simple language — not educationese . 



P-08: Video Tape Modules 



STANDARDS: 



1 04 Quality of voice is animated and speaker did not stumble over 
words, mispronounce. 

1 09 Speaker followed script. 

1 24 Jargon was user relevant. 

1 09 Speaker djd not pause so long it was distractive. 

1 09 No words were missed here or there. 

1 07 Music was adequate (background) . 

1 19 Visuals were adequate and appropriate to topic. 

1 18 Camera shift was appropriate and timely. 

1 06 Reaction of students to tape was positive. 

1 12 Communicates as registered by statistical results. 

1 14 Approval of sponsor. 

1 16 Does it measure up to user's expectations, e.g., coxnmercial TV. 



NO 

01 Determine where training aids (visuals) might be needed in 
conjunction with the presentation. 

06 Suggest corrections in video tape in both format and cont.'"-. 
06 Suggest changes in verbal script. 

06 Edit scripts for accuracy. 

03 Remake video visuals for different (3) response mode (as 
against the 20 response mode).^ 



^One of the research experiments addresses itself to the effect of 
response-demand frequency on learning. To determine this, lecture units 
are made up with different response-demand rates: the high response-demand 

frequency tapes inject 20 questions to the student during the unit (module) 
while the low response-demand frequency tapes inject only three questions; 
the three questions are selected by eliminating all questions except the 
three that occur nearest to the desired intervals through the tape. 



J LM 



TASKS : 
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03 Make sure equipment hardware and software are compatible. 

03 Plan content to suit medium in interesting manner. 

04 Wr itc/prepare script. 

04 Make copy of first script for approval . 

04 Transfer script to teleprompter . 

04 Organize hundreds of s.l ides for visuals (and panel books) . 

04 Transfer video visuals to 35mm slides. 

04 Duplicate video visuals for panel book (for audio tape). 

04 Prepare visuals for appropriate superimposition. 

13 Select reader — someone who knows or is professional reader. 

13 Hiring of subcontractors to care for direction production. 

13 Purchase video tape. 

13 Hiring of talent for performing lecture material. 

13 Select talent personality to suit intended presentation 
realistically. 

53 Contract local commercial TV station to use facilities. 

53 Contract another facility to transfer to one-inch video tape. 

04 Record the video tape. 

06 Edit video tapes for errors of commission and demission. 

06 Detect errors in video tape — both technical and in terms of 
jargon. 

06 Provide comments as to quality of video tapes. 

06 Insure that script was followed by the speaker who was cutting 
the tape. 

06 Edit video tapes. 

06 Edit errors , electronically , as made . 

06 Rewite many scripts for better video contact with viewer, 

06 Eliminate unnecessary big words — be concise, 

04 Get tapes duplicated . 



ENABLERS : 

S 

1 22 Knowledge of audio-visual hardware for media. 

1 04 Knowledge of production techniques relevant to individual media, 
1 17 Knowledge of audio and print writing technique. 

1 04 Knowledge of capabilities and limitations of video as electronic 
medium. 

1 22 Knowledge of how to run video tape machine, 

1 19 Knowledge of user relevant terminology, 

1 05 Knowledge of user procedures and policy, 

2 03 Ability in designing and coordinating video and visual elements 

of presentations. 

2 05 Coordinating all the people involved at one time or in sequence, 
2 15 Ability to read as if he knows material (narrator) . 

2 16 Ability to adapt script, etc, to each medium, 

2 18 Ability to fit everything into scope of medium being used, 

2 20 Ability to make creative decisions. 

2 14 Ability to write for both audio and print. 

3 13 Use of concise, simple language — not educationese . 
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P-09: Audio Tape Modules 



STANDARDS : 

J 

1 06 Product holds interest of students. 

1 14 Approval of sponsor. 

1 12 CoTTununi cates as regis tered' by statistical results . 

1 16 Does it measure up to user's expectations, e.g. commercial TV. 



TASKS : 

04 Transfer sound track from video to audio tape, 

04 Develop scripts complete with instructions to reader re: 
pausing, emphasis , etc . 

04 Get panel book made up to correspond with video visuals, 
04 Duplicate tapes in house. 



ENABLERS : 

S yv 

1 22 Knowledge of audio-visual hardware for media. 

1 04 Knowledge of production techniques relevant to individual media. 

1 17 Knowledge of audio and print writing technique. 

2 15 Ability to read as if he knows material (narrator). 

2 16 Abilit}^ to adapt script, etc. to each medium. 

2 18 Ability to fit everything into scope of medium being used. 

2 20 Ability to make creative decisions. 

2 14 Ability to write both for audio and print. 

313 Use concise, simple language — not educationese . 



P-12: Computer-aided Instruction (CAI) Modules 



STANDARDS : 

J m 

1 12 Statistically good — better than 80/80 performance. 

1 05 CAI module proves more effective than other media. 

1 20 Whether it (CAI module) communicates other than as indicated 
statistically . 
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TASKS : 

EL 

04 Duplicate material for use on other media , as comparison and 
to replace. 

06 Deleted some Irrelevant materials from instructional program. 



ENABLERS : 

1 02 Educational technology as taught by Skinner et al. 



P-13: Linear Programmed Modules 



STANDARDS : 

1 LM 

1 04 Subjective evaluation — appearance of the texts was good. 

138 Subjective evaluation — language of the texts was appropriate 
in relation to the material and target population. 

1 12 Student performance — 80% received scores of 80% plus, 
therefore material is performing its function (teach). 



TASKS : 

01 Review reference materials— textbooks relative to content 
material of the texts. 

03 Correlate coutert outline with materials in the linear texts. 

03 Incorporate material from references into wnriting of the frames. 

03 Select examples, usually relevant, to support material. 

04 Write/create frames . 

12 Confer with subject matter experts as to appropriateness of 
examples to be included in the linear texts . 

19 Provide implementations and supervise alternative ways of 
expressing a particular point in the material. 

21 Follow directives from the user agency for specifications in 
revision of the materials. 

23 Participate in decision of final format/style — input and 
information . 

59 Interact with implementation supervisor for further clarity/ 
specifications in revising linear materials. 

04 Type material in camera-ready form. 

06 Rewrite frames — content within linv^ar text. 

06 Add and/or delete frames. 

06 Revise style of frames — bring paragraphs together; rearrange 
order of frames, add examples. 
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06 

06 


Review/proof-read final materials of linear text. 

Revise sentence-structure/ grammar — does not change the meaning. 


ENABLERS : 
S UV 
2 04' 
2 06 
2 02 
2 23 


A 

Ability to revise frames relative to feedback from testing. 
Ability to write a frame. 

Skill of tact — particularly in dealing with the user agency. 
Ability to persuade or convince of your viewpoint. 



P-14: Intrinsically Programmed Modules 

STANDARDS : 

J LM 



1 22 


Shortest and fastest path through program results from correct 


1 12 


answer . 

Organized so that no response requires turning more than eight 


1 07 


pages either way. 

Intuitive feeling that material covers most probable errors that 
student can make. 


1 09 
1 09 
1 09 


Avoidance of questions that elicit irrelevant responses. 

Avoids too small steps that make material appear trivial. 
Absence of too much repetition — more than necessary for normal 


1 22 


emphasis • 

Elicits reasonable responses after making specific point only 
once or twice. 


1 01 
1 16 


Points to be taught covered thoroughly from several aspects. 
Quality as judged against rules given in "Good Frames or Bad," 


1 12 


a text authored by Dr. S. M. Markel. 

Student statistical performance meets 80/80 — 80% of students 
make 80% correct responses. 


TASKS : 
NO 
05 


Test material with sampling of three to five students at 
contractor site. 


06 

06 


Edit, revise, "patch" copy in rush to meet deadline. 

Apply references from faculty and subject matter to experts 
for revision of textual content. 


06 


Rewrite sections based on deficient student (statistically) 


06 

12 

04 


performance. 

Patch "errors" shown by students making Incorrect responses. 
Retype copy marked up for sending to prlntera. 

Bind and staple copies of programmed texts. 
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54 Textual specifications given and explained to subcontractor. 

54 Examine and critique text (copy received from subcontractor) . 

63 Decide to redesign sequencing for programmed text. 

03 Prepare schematic for producing modified instructional sequencing 
that meets standards of programming, 

11 Train other writers to produce optimumally programmed intrinsic 
programmed texts . 



ENABLERS : 

1 uv 

] 02 Knowledge of ^^process of Preparing Valid Effective Learning.” 

1 04 Knowledge of good programming frames from bad frames. 

2 06 Ability in writing suitable sequences for intrinsic programmed 

learning . 

2 08 Skill in interpreting ^'Preparing Valid Effective Learning.” 

2 06 Ability in producing schematics for scrambling programmed sequences. 
2 01 Experience as military instructor. 

2 14 Ability to avoid contradictions or apparent contradictions in 

written' material . 

3 30 Ability to think like a student. 



P-16: Overall Research Design and Evaluation Plan 



STANDARDS : 

J m 

1 16 Compares favorably with other efforts experienced outside of 
present agency. 

1 05 Quality enhanced by virtue of attending to a long term unit of 
instruction where little work has been done. 

1 09 Data gets into all the cells of the research design and comes 
from the right subjects. 

1 21 Variables which contaminated earlier efforts were uncovered 
and controlled . 

1 08 Classification of test items (acquisition or application) was 
confirmed by others. 

1 22 The run of the course was able to follow the chart set by 

the research design and adhered essentially to the conditions set. 

1 23 Successfully constrains instructional implementation to 
research design specifications. 

1 23 Should be changes in content when system design is used. 



Examine all objectives of all segments of the course for the 
purpose of creating a "content map.” 




TASKS: 

m 

01 
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01 Develop rationale for requesting an unscheduled run of the 
course. 

01 Read all back reports to prepare to handle research and 
evaluation coordination after two years of operation. 

01 Study the implementation aspects of the first run of prototype 
course materials (Phase 1) . 

01 Study and develop hypotheses about possible subtle effects of 
slight variation in instructions to students taking course. 

02 Confer with users and developers rating the course on how best 
to give users a sense of the integrated wholeness of the course. 

02 Develop hypotheses about various levels of information which could 
be given students about the various experiments-, conditions used. 

03 Confer with consultant on guidelines for determining when effects 
of learner characteristics could be examined with respect to 137 
characteristics listed, 44 students in course. 

03 Develop general specifications for implementing a confirmation 
answer sheet to student managed progress check materials. 

03 Confirm sequencing of instructional objectives by consulting 
with content specialist. 

03 Develop a plotting chart for all segments of the course, including 
all variables manipulated within. 

03 Assign random selection of students to cells within the plotting 

chart, sequencing various groups differently through the variables. 

03 Confirm correct treatment label of each module in terms of the 
instructional condition within . 

03 Confirm that the number of groups plotted corresponds to the 
number of groups necessary to address research questions. 

03 Confer with consultant with regard to the need to ''force'* some 
students through materials previously routed around. 

03 Provide materials production personnel with specifications for 
materials to be developed based on research questions. 

03 Provide materials production personnel with specifications for 
developing and implementing test items to establish norms. 

03 Design parallel modules using identical presentation in different 
media • 

03 Vary response demand and management frequency as defined by 
research design. 

04 Rewrite terminal objectives for each instructional unit into 
content statements appropriate for a content map. 

05 Maintain record of instructor-student interaction time (tutoring). 

05 Maintain record of total time instructor involved within 

instructional system. 

05 Maintain record of time required for course administrator 
(materials control clerk). 

05 Assess from data which media forms in the instructional materials 
appear to be most effective, liked by the students. 

05 Tryout initial instructional unit prototypes on small group 
of students to assess operability, etc. 

05 Analyze data from tryout to determine required revisions of 
instructional material. 

05 Review with field-setting instructors any problems they 

experienced in the first experimental and developmental run of 
the course. 
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06 Kxamine ccmput«*r doc\iments utilized in course to determine 
refinements necessary to accommodate course users equipment 
likely to be found in a broad range of settings, 

06 Determine, from Phase 1 data, a tentative optimal frequency 
for building“in response sensitive feedback in instructional 
materials . 

06 Update research design to incorporate an examination of the 

effects of use of confirmation answer sheets vs. no confirmation. 

06 Update research design to incorporate an examination of the 
effects of use of content maps on student performance. 

06 Confer with instructors about possible discrepancies between 

intended implementation of the course and actual implementation. 

06 Recommend printed version of all media dependent materials be 
included in research design if such backup is produced. 

06 Update research design to demonstrate effects of use of 
identical items iu pro* and post-criterion tests. 

06 Determine data contaminated as a function of any incorrect 
implementation procedures in Phase 1. 

06 Determine need to add an unscheduled run of course to obtain 
appropriate data for research questions not answered. 

06 Review with field setting all test items for appropriateness 
and whether each represents an acquisition or application 
performance . 

06 Review student questionnaire responses and interpret in light 
of student performances in the course. 

12 Monitor personnel who are documenting cost factors involved 
with development, implementation, and operation of course. 

12 Delegate task of refining of instructions in use of instruc- 
tional materials to others. 

19 Interact with instructors and faculty supervisor regarding 
use of identical items in pre- and post-criterion tests. 

19 Interact with field setting to obtain the required cooperation 
in implementing course rigidly enough to support research. 

19 Make clear the manner in which each student is to be sequenced 
through various media to support the research design. 

19 Make clear to the field setting the necessity for student 
response modes being limited only to that specified. 

19 Review with field setting all course implementation instruc- 
tions including specified paths for specified students. 

19 Explain to students and instructors the rationale for not 
using response sensitive feedback in the early development 
of materials. 

19 Confer with instructors about placing trust in the student 

in developing the strategies for response sensitive feedback 
materials .. 

21 Participate in staff discussions regarding the purpose, intent, 
and content of reports to be written. 

21 Participate in staff and user meetings regarding size of instruc- 
tional booklets and relative costs for ongoing utilization. 
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ENABLERS : 

S \N 

1 03 Know one statistical design from another and what the impli- 
cations and assumptions of each aro. 

1 03 Knowledge of the meaning and value of behavioral objectives. 

1 04 Knowledge of how much one can do with a computer in assisting 
and/or managing instruction. 

1 03 Awareness/knowledge of the typical problems which can beset a 
research design and confound the data. 

1 03 Knowledge of the rationale behind various statistical designs. 

2 02 Ability to respond to questions or criticisms that convey an 

appreciation of the concerns expressed . 

2 02 Skill in sorting out and maintaining a consistent posture in 

relation to where each area of expertise resides. 

3 16 Sensitivity :o experimental design rs a means of addressing 

relevant questions. 

3 11 Sensitivity to instructional systems as an important construct 
in developing technology in education. 

3 21 Sensitivity to the minor details that can contaminate or 
confound the real meaning of collected data. 

3 03 Sensitivity to the needs of teachers and how those needs can 
confound developmental efforts if not attended to. 

3 18 Sensitivity to test construction that guards against self- 
fulfilling measures. 

3 14 Sensitivity to reality and what is possible to achieve within 
the limits of a project. 

3 16 Can’t be perturbed or easily upset by those who cannot "see 

the obvious" (understand the need for purposes of research design). 
28 V^illingness to admit to rigidness in one’s self and to give a 
little bit in the face of an impasse. 

3 10 Sensitivity to identifying those critical variables in a research 
effort which should be least confounded by the design. 

3 12 Willingness to fill "crecibility gaps" between producers and 
users as they are identified. 

3 03 Sensitivity to the need for developers and consumers to start 
on a common base or framework. 



P-17: Instructional Presentation Design 



STANDARDS: 

J m 

1 08 Agreement among group administering course that design is 
adequate. 

1 05 Utility of method for running course (it seem^ easier). 

1 06 Satisfaction (lack of complaints) of student with method. 

2 04 All materials ready on time, in right place. 
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TASKS : 

02 Work with consultants about objectives of presentation design. 

03 Establish schedule for students taking tests. 

04 VJrite instructions for students’ use of each instructional 
segment . 

04 Get all course segments and testing materials ready for next 
run . 

04 Write new instructor guide. 

04 Create methods to keep track of all data (control systems — 
student data file), 

04 Develop form for suggested revisions. 

05 Keep records of materials which were handed out (used). 

05 Do hand analysis of data to determine percent of correct 

responses to each test question. 

05 Use (administer) form for suggested revision. 

06 Help change research patterns for fall to ones which will be 
possible in classroom. 

06 Work out new processes for course administration. 

07 Hand ovit tests and materials. 

12 Schedule material from development people on time for fast 
students . 

12 Schedule activities for next run — flow chart of activities 
geared to calendar. 

19 Explicate students' responsibilities out of research design. 

19 Define role of instructor, course administrator, student. 

19 Motivate students to not sabotage research. 

21 Interact with other responsible persons on course administration. 



ENABLERS : 

1 uv 

1 07 Knowledge of research design of this project in order to 

correlate instruction efforts with objectives of research. 

1 07 Knowledge of course pattern (not content) to interpret to 

students. 

2 17 Skill in reading computer printouts about student performance 

in order to alter course structure. 

2 05 Ability to use, in proportion, time for scheduling. 

2 17 Skill in flow chart reading, to use in following directions 
of research staff. 

2 05 Skill in conceptualizing flow chart to direct students. 

2 33 Skill in flow chart preparation to direct students. 

2 23 Skill in motivating, in working with students to keep course 
moving. 

2 13 Ability to listen carefully to complaints of students in order 
to modify course, help student. 

2 21 Ability tc organize, to keep track of all parts of course 

and materials important to each. 

3 22 Sensitivity to when students were sabotaging research to 

help evaluate data. 
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P-22: Progress Checks 



STANDARDS ; 

J ^ 

1 01 Test items cover the material included in instructional materials. 
1 04 Test items are not ambiguous. 

1 19 Distractors seem to be logical distractor test items. 



TASKS : 

NO 

04 Write test items to produce progress check keyed to the 
instructional materials. 

06 Read instructional materials that test items covered to insure 
correspondence between test items and Instructional materials. 

06 Check terminal objectives/enabling objectives to see that they 
were covered by test items. 



ENABLERS : 

1 08 Knowledge of the subject matter/content area related to 
progress-check test items. 



P-24: Student Data File 



STANDARDS : 

J m 

1 22 All data filed so it can be found. 

1 16 Easier tracking of data, as felt by group, than earlier me;_hod. 
1 06 Agreement that process does what is intended. 

1 13 Agreement by research director that method doesn't compromise 
research. 

1 09 Lack of trouble in data processing. 

TASKS : 

NO 

01 Figure out what is information needed by research instructor 
in way of student data file. 





F 
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Make chart of modules, tests, deadlines, student assignments. 
Keep records on materials handed out. 

Keep records on course segments completed by students. 

Invent methods for handling progress and cumulative test data. 
Show exact units received by each student. 

Show scores made by each student, either failure or pass by 70/u. 
Check questions missed to give specified remediation assignment. 
Keep record of the delinquency status of "laggers." 

Enter test results on student progress chart. 

Compile record of location of all students at time of test. 

Check for students using other than assigned media. 

Gather and send data to data processors. 

Do hand analysis of data to computer percent of students 
responding correctly to each test item for use in improving 
course. 

Keep duplicate records of all data. 

Encourage students to take research as well as grading tests. 

For second failure, refer for tutoring — follow up to see 
what student gets. 

Communicate with students through "mail boxes," 



ENABLERS : 

i 

1 07 Knowledge or course administration design, 

1 07 Knowledge of research design of this project in order to keep 
data properly, 

1 03 Knowledge of hand-sort methods in order to select the required 

data for each research concern. 

2 10 Ability to organize in order to develop methods for storing 

and retrieving data. 

2 11 Ability to take care and patience in record keeping in order 
to reduce errors in data. 



I 



OA 

OA 

OA 

OA 

05 

05 

05 

05 

05 

05 

03 

05 

05 



12 

19 

20 

21 



P-25; Student/Instructor Use Guide 



STANDARDS: 

J 1^ 

1 23 Students use guide — questions reflect understanding of manual 
information. 

1 13 Approved by project manager, program manager. 



O 

ERIC 



TASKS : 
01 



Review all related course materials. 
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01 Describe the traditional course that the new mediate course 
is paralleling. 

03 Build outline of total use guide encompassing all of the above. 

04 Clearly state, in writing, instructor's role. 

04 Put in history of the course in leadership at the user agency. 

04 Point out that helping research contributes to user leadership. 

04 Describe student role in mediated course. 

04 Write total guide, using outline as basis. 



ENABLERS : 

S UV 

1 07 Knowledge of history of the project. 

1 07 Knowledge of research design of project. 

1 07 Knowledge of new mediated course functioning in terms of what 

is expected of instructor and student. 

2 18 Ability to sense fitness of material for sponsor's target. 

2 14 Skill in writing clearly so someone unacquainted with the 

course can understand the roles of instructor and student. 

3 03 Sensitivity to needs of students. 

3 03 Sensitivity to needs of instructor. 
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P-26: Course Development Model 



STANDARDS: 

1 17 Is the same as or as good as material on regular course as 

judged by leadership group. 

2 23 Course has changed from regular course, includes explanation 

of why change. 

1 17 Each part of unit fits within meaning of course outline, 
covering some of the necessary objectives, 

1 0/ Each segment of writing is acceptable to me in terms of 
content . 

1 11 Segment of course meets criteria in research design. 

TASKS: 

NO 

01 Analyze all data about traditional course content. 

02 Talk with agency colleagues about military needs for course. 

02 Build objectives from course outline, one terminal objective 

for each major segment, two or more enabling objectives for p.ach. 

03 List new suggested course content. 

04 Rebuild course outlines pn basis of <=ponsor reaction to 
suggested list. 
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04 Work with colleagues on final draft of content outline, 

04 Finish writing Part 5 for Phase 1 final, 

06 Delete parts of course and related objectives to make content 
outline short enough for sponsor to accept, 

07 Negotiate with sponsor representatives about acceptance of 
components , 

07 Negotiate with sponsor about acceptance of components. 

07 Get total new course content outline approved by sponsor, 

12 Work with agency consultants to work out course content outline 

for Part 5 to include all specified subjects and develop terminal 
and enabling objectives for each, 

14 Present recommended course content to sponsor for reaction. 

20 Provide required course content information from sponsor to 
project. 

21 Interface with individual students when there are problems, 
or when getting material, briefing, etc. 

21 Provide input re: affect, experience of teaching unmediated 

course to project, 

21 Work with writers to write parts of course. 

21 Work witl. writers about Part 5, developing course materials 
from objectives. 

21 Work with consultant about writing outline for strength in 
teaching patterns, 

21 Work with writers about content (sponsor examples) units of 
component . 



ENABLERS : 

S W 

1 08 Knowledge of previous course content. 

1 07 Knowledge of project research design. 

2 02 Skill in negotiating with sponsor. 

2 30 Ability to be flexible when decisions changed by sponsor. 

2 07 Skill in teaching to use in teaching traditional course as 
it has been taught to compare with experience of teaching 
new mediated course. 

2 02 Ability to work with writers on course content changes from 
sponsor (interpreting) , 

2 02 Ability to explain and negotiate with sponsor the course 
content changes from pro j ect . 

2 11 Ability to do things over and over without guidelines until 
they are acceptable . 

2 11 Ability to tolerate criticism and rejection of work without 

losing desire to start over and do it again. 

3 16 Sensitivity to military standards : morals , fitness , duty , 

ideas of leadership. 

3 16 Sensitivity to what sponsor would complain about in examples 
in course material. 
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P-28: Master Tutor Modules 



STANDARDS : 

J 1^ 

1 06 Product holds interest of students. 

1 14 Approval of sponsor. 

1 12 Coinmunicates as registered by statistical results. 

1 16 Does it measure up to user’s expectations, e.g., commercial TV. 



TASKS : 

m 

04 Duplicate intrinsic programmed unit on 4-track tape. 

04 Program tape for computer operation in response to student 

selections . 

04 Get tapes duplicated . 

13 Get readers for script . 



ENABLERS: 

^ W 

1 22 Knowledge of audio-visual hardware for media. 

1 04 Knowledge of production techniques relevant to individual media. 

1 17 Knowledge of audio and print writing technique. 

2 15 Ability to read as i f he knows material (narrator). 

2 16 Ability to adapt script, etc. to each medium. 

2 18 Ability to fit everything into scope of medium being used. 

2 20 Ability to make creative decisions. 

2 14 Ability to write for both audio and print. 

3 13 Sensitivity to using concise, simple language — not educationese . 



P-29: Structural Communication Module 



STANDARDS : 

J 1^ 

1 04 Material understandable and easy to follow. 

1 17 Material is in correct logical sequence. 

1 09 There were no discrepancies in the information presented. 
1 04 Instructions wore clear/understandable. 

1 01 No major paragraph/page iiissing — materials were complete. 
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1 19 Subjective evaluation — examples were appropriate, and il.ei*:: 
were enough of them. 

1 05 Material is appropriate for targec audience (midshipmen), 

TASKS : 

m 

01 Review military leadership publication for purpose of 
extracting examples and case studies. 

03 Design of structural communications units by subcontractor. 

03 Provide subcontractors with basic content materials for units. 

03 Provide specific instructions as to how to use a specific 
phrase relative to "American" language. 

03 Provide military e.xamples around which units could be created 
or constructed. 

04 Rework materials to insure correct format. 

04 Write the structural communications units. 

06 Edit peculiar British phraseology in matrix statements. 

04 Package material in booklet. 

05 Do the lesson within the structural unit(s) — work through 
material . 

05 Field test materials with target population types. 

05 Record student comments relative to materials. 

06 Review/edit the unit material in a structural communication unit 

06 Read materials to insure understanding of content and instruc- 
tions (trial run) . 

06 Modify unit material on basis of field test with target 
population. 



ENABLERS : 

1 UV 

1 08 Knowledge of American military leadership (user relevant) 
situations . 

1 19 Knowledge of military (user relevant) terminology. 

2 18 Ability to detect differences in meaning between item in 

response indicator and the author’s rationale for the item 
for purpose of revising. 

2 08 Ability to thoroughly read through materials to note any 
errors , misunderstandings , discrepancies . 



P-30: CBI-70 Modules 



This is a potential form of module that had not, at the time of 
observation, been used. The CBi70 machine on which the film V7ould be 
run, if modules are developed for it, was at th- project with a demon- 
stration film. How’ever, as no modules have been made, no standards 
and enablers are listed for it. 
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STANDARDS: 

No information collected under this heading. 



TASKS : 

04 Program video and panel-book material to duplicate intrinsic 
test . 

04 Produce 6-track film with cuing. 

04 Get films made and printed. 



ENABLERS : 

No infornuition collected under this heading. 



P-42: Content Map 



STANDARDS : 

J m 

1 13 Acceptance by researcher. 

1 07 Check by self for completeness. 
1 18 Check by self for good looks. 



TASKS : 

03 Draw meaningful phrases from course outline enabling objectives 
and terminal objectives to give cues to students about what it 
is expected they will learn. 

04 Display in flow chart the sequence of segraents and relationships. 
04 Display terminal objectives and enabling objectives. 




ENABLERS : 

i 07 Knowledge of course structure in order to display it. 

1 03 Knowledge of what is meant by enabling and terminal objectives. 

2 05 Skill in flow chart techniques for drawing flow chart of 

course progress. 

2 14 Skill in writing, for phraseology, clarity in explaining 

objectives to students . 

3 03 Sensitivity to students’ needs to visualize toual course 

section. 
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Appendix B: Details on Each Management Responsibility 



In Chapter III, the management responsibilities are divided under 
two headings. Production Management and Environmental Management. On 
this project, most of the management information obtained was production, 
as personnel whose primary responsibility was at the environmental 
management level did not receive formal interviews, although they pro- 
vided useful contextual input. 

As the methodology was in course of revision at the time of observa- 
tion, the data that follow were collected under one management heading, 
which is essentially more in the realm of production management than 
environmental management, although separate categories are given. 



Listing of Standards, Tasks, Enablers . 

The following are the lists of management responsibilities, using 
the same format as in Appendix A: 



PM- 35: 


Instructional Materials. 


EM-36: 


Budget . 


PM-37: 


Contract (Proposal) 


EM- 38: 


Staff Morale. 


EM-39: 


Interagency Cooperation. 


PM-40: 


Product Quality. 


EM-41: 


Staff Hiring. 


EM-43; 


Intra agency Cooperation. 


EM-44: 


Project Accountability. 
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PM-35; Instructional Materials 



^TANDAT’.DS : 

J 

1 07 Meets own set of standards for quality. 

2 04 Meets schedule for timelines. 

1 01 (Includes all items on chart of instruction package, i.e., 
syndactic text, video, etc,). 

1 07 Meets own check list for completeness. 

1 13 Meets editor criteria. 

1 14 Acceptance of each segment by sponsor. 



TASKS: 

m 

52 Plan production and delivery schedule to sponsor. 

52 PERT production schedules back from delivery date to cover 

work to printer, typing, proofreading, incorporation of 
student evaluation data, rewrite, and original writing. 

54 Supervise clerical help for promptness, quality. 



ENABLERS : 

S W 

1 08 Knowledge of the course content. 

1 08 Knowledge of the course objectives. 

1 08 Knowledge of the course content outline. 

1 07 Knowledge of sponsor standards. 

1 07 Knowledge of course development total package. 

1 03 Knowledge of test item writing for preparing questions for tests. 

1 05 Knowledge of military leadership to use in preparing examples, 

2 19 Skill in planning production schedules. 

2 05 Skill in organizing manpower and resources to get things done 
on time. 

2 34 Skill in coordinating various activities as they support each 
other . 

2 29 Ability to work with writers to help them understand and meet 
standards . 

2 35 Skill in language to use in editing and rewriting sections 

of texts so that they are clear, informative. 

3 01 Sensitivity to likes and dislikfjs of sponsors/users. 
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EM-36: Budget 



STANDARD^! 

J IM 

1 16 This budget similar to previous budgets. 

1 16 This budget compares favorablely to budgets of similar projects. 

1 13 The boss accepts and approves the budget. 

2 11 Actual expenses fall within estimate figures. 



TASKS : 

NO 

01 Review previous budgets for form and figures. 

12 Estimate new figures for budget. 

12 Determine fixed costs of project. 
i2 Estimate nonfixed costs of project. 

12 Readjust nonfixed costs to meet budget limitations. 

14 Submit budget to accounting office for approval. 



ENABLERS : 

S W 

1 04 Knowledge of accounting procedures. 

1 19 Knowledge of accounting jargon. 

1 16 Knowledge of accounting forms. 

1 11 Knowledge of accounting purposes. 

1 11 Knowledge of accounting taxes, depreciation, etc. 

1 11 Knowledge of accounting company policy. 

1 12 Knowledge of whom to go to for specific information. 

1 01 Knowledge of matl ^atics. 

2 13 Ability to listen to experts for suggestions, 

2 26 Skill in assessing the sources of knowledge. 

2 27 Skill in use of a calculator. 

2 22 Skill in estimating expenses. 

2 04 Skill in using current operating expenses for projection . 



36o 
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PM-37: Proposal Contract 






standard?; 



No information collected under this heading. 



TASKS : 



m 

02 Provide guidelines for the hypotheses to be tested in lieu 
of actual hypotheses . 

03 Specify research model to be employed in coordination with 
instructional materials development. 

12 Specify those items (products) to be delivered during 
period of contract . 

12 Specify to whom the products produced are to be delivered. 

12 Specify the time schedule for delivery of products from 

contractor to contractor . 



KNAHLERS : 

S W 

3 03 Sensitivity to needs of the educators. 
3 03 Sensitivity to needs of R & D centers. 



EM-38: Staff Morale 



STANDARDS : 

No information collected under this heading. 




TASKS : 

m 

15 Held weekly conference with strff on Saturday morning to 
discuss rumors and piece out reality from fantasy. 

14 Conduct Saturday staff meetings to j,lan strategy for next 
week — we’re going to do this; if that doesn’t work, we do 
etc. (determine alternatives). 
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15 Explain the environment, responsibilities — then get out of his 
(staff’s) way. 

19 Determine that memos written by either agency are acceptable 
to both through conference with other agency manager. 



CTABl.ERS: 

S W 

2 23 Skill in keeping them going when it got tough. 

3 19 Sensitivity to needs of staff for support. 



EM-39 : Interagency Cooperation 

( 



STANDARDS : 

No information collected under this heading. 



TASKS : 

m 

05 Create a program data file for maintaining all data generated 
during instructional program development; 

19 Make suggestions to program manager (other agency) on how his 
activities relate to this project . 

19 Hold weekly lunches with other agency manager to discuss ways 
of dealing with problems jointly. 

19 Translate jargon of experts into language understood by customer. 

2G Precipitate meetings with sponsoring agency through contract 

manager. 

20 Act as interagency gatekeeper—assemble all of the information 

relative to progress at one point; establish a communication 
center. 

20 Inform about and involve sponsoring agency in only those items 
which promise greatest yield-process module, etc. 

21 Maintain regular meetings with sponsoring agency. 



ENABLERS: 

S ]JV 

1 10 Knowledge of interpersonal relations both within other agencies 

and between agencies. 

2 12 Ability to insist that only the project manager makes commitments 

for project tasks. 




3 16 Sens! .ivity to value conflict — start with assumption that 
contracting agency is hostile environment. 

3 16 Sensitivity to modus operandi of contraction group — these 
people (miMtary) have a working relationship with people 
whicn i:; an adversary kind of procedure. 

3 02 Sensitivity to personal limitations — is all the cunflict, 
strife, anguish really worth the effort. 

3 13 Sensitivity to language problems — make sure that the 

interface between customer and expertise is done at one 
point; there is one communicator between agencies. 

3 13 Sensitivity to need for gatekeeper between expert and customer, 
to avoid semantic misinterpretation previously settled by 
gatekeeper . 

3 18 Sensitivity to problems of multiple losses. 



PM-40: Product Quality 



STANDARD? • 

J 

1 12 Products of project meet performance criteria. 
1 06 Product accepted by user. 



TASKS : 

m 

06 Examine instructioaal material consent for accuracy, 

06 Examine processes of instructional unit for weaknesses which 
violate assumptions 'Jn estimating the effects of the unit. 

06 Examine instructional materials format and processes for 
affective impact on students. 

12 Restrict (critique) commentary of contracting agency to that 
relevant to content of material submitted and compatible with 
rest of curriculum. 

14 Submit copy of completed material to contracting agency and ask 
for comments . 

14 Resolve conflict of quality control standards between contractor 

and contractee on such factors as grammar, quality of visuals, etc 

14 Write out quality control procedures and provide quality control 
checklist to contracting agency. 

14 Involve students in the decision process as third party mediator 

when other agencies arc in conflict over quality control standards 

14 Examine instructional material content for continuity and un- 
planned or unnecessary redundancy. 



i 
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14 Negotiate instructional materials performance criteria with 

respect to percent of students to succeed at what criteria level. 

14 Direct revision of instructional units in accordance with 
need to strengthen various elements. 

14 Specify weaknesses of instructional units to provide the 
basis for revision efforts. 

21 Hold monthly meetings with head of contracting agency 
section .o discv-ss quality control problems. 



ENABLERS ; 

s m 

1 07 Knowledge of what contracts state as products required. 

3 15 Sensitivity to realize that quality controls are negotiated 
standards . 

3114 Sensitivity to the degrees of freedom within contractual 
obligations — how much variation we intend, low much they 
will allow. 

3 01 Sensitivity to the point of extremity — where does one take a 
stand--where one feels the push is beyond the limits of 
contractual variance. 

3 18 Sensitivity to being pushed into something you shouldn’t be 
doing — going beyond contract. 

3 02 Sensitivity to human limits — confirming the work to 
contracuual task statements • 

3 13 Sensitivity to language problems — making sure you and tne 
customer do not have any semantic problems, schedule 
problems, or misunderscanding of task statement problems. 

3 16 Seasitivity to attitude of taxpayers. 

3 16 Sensitivity in quality control negotiations — to whether 
we have enough leverage to force our standard or whether 
it is really that important. 



EM-41: Staff Hiring 



STANDARDS • 

J 

2 20 The kinds of questions the person (candidate) asks are 
considered appropriate. 

2 20 References indicate candidate works well with other people 
and he can learn. 

2 20 Candidate is a rapid learner, knows the key questions, and 
doesn’t waste my time. 

2 14 Candidate impresses me favorably in comparison with my 
colleagues in previous sitUtitions. 



7 5 
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TASKS: 

NO 

1] Hire only thos« people you consider exceptionally good, 
generally with at least one specialty area. 

11 Keep file of resumes and track good people in their careers — 
keep in touch at conferences, etc. 
li Pick up the phone and call everyone yon know to advertise 
job opening and inquire about availability. 



ENABLERS : 

s uv 

1 12 Knowledge of petople v;ho are in educational research and 
development business. 

1 12 Knowledge of agencies that do educational research and 
development . 

1 12 Knowledge of specialties of individuals. 



EM-43: Intraagency Cooperation 



STANDARDS ; 

J IM 

1 22 Products of project meet performance criteria. 

2 04 Products are produced and delivered on schedule. 

2 13 Products are produced within budget. 

2 05 Staff produces acceptable products as individuals and as a 
group . 

2 25 Staff interacts as a group a constructive manner and with 
mutu^il respect. 

2 25 Staff views each individual member as being capable and 
competent in the performance of his tasks. 

2 26 Staff reflects sense of values and priorities compatible with 
project objectives . 

2 08 Staff reflects a faith in the judgment of project leadership. 

2 27 Outcomes of decisions made in tense situations result in 
constructive activity following . 



o * 
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TA8KS : 

NO 

15 Convey to production staff that their biases and concerns 
are fairly represented in negotiating characteristics of 
a uniu of instruction. 

15 Assess productivity of project staff to support reconmien- 
dations for salary increases. 

19 Negotiate a viable philosophical basis for development of 
instructional materials between groups holding opposing 
value systems. 

19 Attend meetings between agencies to annotate the relative 
positions of each regarding course development procedures, 

19 Identify tha nature of the differences of opinions between 
agencies relating to course development procedures. 

19 Adapt course development procedures to reconcile differences 
of opinions coming from different value systems. 

19 Stress to production staff that the focus is on end product 
in order to minimize '*conf rontation** aspects of review and 
refinement . 

21 Confirm in writing agreements reached in interagency meetings 
which identify the next steps to be taken. 

21 Confirm agreements reached in interagency meeting which describe 
the actions to be taken as a result of that meeting. 



ENABLER/^: 

No information collected under this heading. 



EM-44 : Project Accountability 



STANDARDS : 

J 

1 22 Products of project meet performance criteria- 

2 04 Products are produced and delivered on schedule. 
2 13 Products are produced within budget. 



TASKS* 

NO 

11 Estimate number of personnel required to deliver products 
within time frames available. 

11 Interview applicants or desired persons with respect to 
project employment . 



832 



12 Determine budgetary allocations for salaries > materials » 
services > etc . 

12 Determine the schedule of contracted delivery of products 
to consumer* 

12 Determine time factors involved with reviev’ and refi;iement 
of instructional materials . 

12 Making explicit what products must be produced. 

13 Interview/check with potential consultants who might be 
able to produce critical products quickly. 

20 Give agency rationale for moving course development staff 
to site to facilitate direct interaction wit': consumer. 



ENABLERS : 

No information collected under this heading. 
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Appendix C: Glossary of Abbreviations Used on Project 

CAI CompuLer-aided instruction: a type of module. 

CBI70 A computerized film learning machine. 

CMI A facility in Iowa used for processing data. 

CPT Cumulative Post Test. 

HMF High Management Frequency. 

HRDF High Response-Demand Frequency. 

IP Intrinsically Programmed (module). 

LAS Learning Activity Summary. 

LMF Low Management Summary. 

LRDF Low Response-Demand Frequency . 

medMF Medium Management Frequency. 

MF Management Frequency: frequency within segment at which response 

causes management to vary stimuli fed to student. 

PC Progress Check. 

RD Response Demand: occasion when student is expected to respond 

to a question. 

RDF Response Demand Frequency: occurrence of response demands 

within a segment. 

SC Structural Communications : 

SME Subject Matter Expert. 
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Chapte r I : 



0 ^ervlo'; 



The overview presents a brief synopais I'f the AIMS Project as an 
introduction. This is elaborated bv a discussion of objectives, ratitaiale, 
and si^nificanee of the project and the* context in which the project operates. 

S ynopsis of the Project 



Responsible Institution: Lo.s Angeles Unified School District (L\USr). 

Subcontractor: Systei^s Developnon Corporation (SOC) . 

Funding Sources : 1 . LI . S . Office of education , National Center for 

Educational Research and Developnent. 

?. Los Angeles Unified School District. 

Funding Duration: March 15, 1970 to March 17, 1971. (12 months) 

Observation Date: December 1970. 

bresent Stage ot Development: Entering final third of project. 

RDD&E Focus of Project: Educational developnent. 

Focal Product: Design specifications and implementation plan for 

an automated instruct ional-ma ter ials-handling system. 

Level of Funding and Duration: ?!edium-Low. (level 3 of 7 levels) 

Agency Setting: Public school district (large metropolitan) and 

private industry . 

Staff Summary (Current): Professional Suppr-r 

Total Full Time Equivalency (in man years): 3.85 1.00 

Number of Personnel Assigned : 



Title: Study and Dev lopnent of Automated Instructional Ma tctrir.l 

Handling Prograr . 



Prime Contractor 
Sub contractor 



3 

5 



1 



Professional Specialities of Staff (interviewees only): 

library science (3), information science (documentation), 
education , language ^ business administration , and 
advertising and marketing. 
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‘ l! Hat Iona Us r 1 f 1 c ancty o f J* j < ' 

The first goal of the Study and Development cf Automated Instructional- 
Mater i als-llandling Program (AIMS) is to design an automated systeiM for 
supplying selection, storing, dis t'" ibution , purcha^Ung, controlling, aivl 
evaluation information about instructional materials to meet the bos 
Angeles Unified Sr lool DistrlctMs (LAUSD) needs for an integrated and 
ef f ic ien t ma terials-[^andl ing program . 

A second goal is “to plan and design the system in suci^ a way t 
t can, with m'nor alterations, be readily adapted for use in otl^et 
California ^cbjol districts and in school districts in other Great Cities.” 

To support these goals a comprehensive analysis of the systems and 
procc lures in LAUSD, in other California city school districts, and in 
other Great Cities is being conducted to evaluate the systems in use and 
to assess the needs of the present and potential users of instructional 
materials. 

Due to the great increase in the volume , diversi ty , and potential 
application of instructional materials, as well as the increased demand 
for them in school districts throughout the country, the problem of 
managing these materials has become critical. All of these materials must 
be selected, purchased, controlled, and evaluated. The diversity of these 
materials requires that each type receive some form of special treatment, 
while most of the management activities are similar and often quite 
repetitive. Consequently, to meet the increased demands for instructional 
materials of all types , schorl districLS have allocated more of their 
financial and administrative resources to acquisition of these materials . 

The need for distribution of the materials and inforr^ation about them has 
resulted in different systems, some sophisticated and others relatively 
crude, being used for different types o.^ materials. This diversity of 
ways in which they are acquired stored, and disseminated lias Intensified 
the information management problem. 

The basic problem to which this project addresses itself is ^finding 
the most efficient, effective, nonredundant means for managing instruc- 
tional materials in a manner that both satisfies their users and brings 
order to the management process. . (see Footnote 1). 

This project’s significance is in: (£) its attempt to design a com- 

prehensive, automated system for providing essential management information 
(selection, purchasing, controlling, storing, distributing, and evaluating) 
on both book and nonbook^ instructional materials for more efficient 
handling of information about such materials; (^ its attempt to design a 
prototype system that can be adapted for use throughout other school dis- 
tricts in the country . 



^ Cited from the subcontractor’s Technical Support Proposal. 

Nonbook instructional materials vary from audio-visual tapes to 
frogs for science labs. 

,2 -.cv; 
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l.ach rit'* and t*a<’h school tli.srriri iias a \mimic svston that rol'li'('ts 
the prohloKis iml4i;inoiiS to Its local, State, niui oven rop.ional sottiii)’. 
ilowevor, cMTtaln nct’vltU*s and r(’qu i r(‘non ts throii^dnnjt tlu* library and 
Instructional mat(»rlals service centers nrc' sinllar and will hiMU-tlt fror. 
the es tab 1 I ‘diri(‘nt oT stand/ird methods. I'hls s tandard I zat ion could 
facilitate incrensi'd cooperative activities annnv; sc!u>o] districts In 
exclian^; I ne, acquisition and evaluation i n f<m’ia t i ()n , and per’aatis even the 
nnate rials theriselves. ^ 



t>on t^ I n^ 1^1 1 ch tlu * rr oject Operate s 

R cyki t J qi i_s I lip to other agencies . I’ r o j tu* t AIMS resides in 1 1 u‘ I M b r a r y 
S(*rvlces .section of LAl'SD. The director of tliat section is also directing; 
tiie proji’ct, assisted by LAUSD Library Services staff personnel, and a local 
Rlannlny, cird Cuidanct Panel (PGP). The Library and Documentation Systems 
Dtfpartment of the Pul)]ic Systems Division of the Systems Development Cor- 
pf.iation (SDC) is providing the t(‘chnlcal support work for the project 
tlirough a subcontract with LAUSD. 

Liaison is maintained with a National Ad\^isory Committee and a joint 
committee of the California State Library and AutI io-Visual Association. 

'Phe National Advisory Committee consists mostly of selected representa- 
tives from otlier Great Cities. PGP is made up of the Directors of Data 
Processing, Library Services (the Project Director), and Audio-Visual Serv- 
ices for the school district, along with other members of the school dis- 
trict staff. They monitor project progress to insure representation in 
the system under design. Liaison with the state and national committees 
is intended to insure the applicability of the system under design to 
school districts generally. Figure 1 illustrates these project contexts. 

The project receives 86Z of its total funding from the U.S. Office of 
l^iducation (USOL) and 14L from state (California) sources. 

R elationship to other efforts of an overall program . The pro jec t 
herein is the Phase I effort of a projected D^;o-phase program of system 
design and implementation. The Phase TI effort, wlien funded, will be 
tlie implementation of tlic information system in the Division of Special 
Services of LAUSD. 

Supporting and technological resources , Technical skills vital . 
this project are being provided by the subcontractor. These skills are 
in the areas of infonr.ation science and library systems, and application 
pi systems analysis and design (specifically to automated library operations, 
as well as the whole range of instructional materials management). 

fine lines . Work schedules and tine lines for their accomplishment 
are relevant to subcontractor operations. Monitoring of tnis progress 
is done by prime contract personnel. Figure 2 illustrates the established 
time lines for the 10 mjst signiticant production tas'*' , as viewed by the 
pro j ec t . 



3 This description is based on tl\e subcontractor’s Technical Support 
Proposal . 
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FLG. Project time lines chart (for subcontractor). 



Descriptions of t])e subcontractor's project tasks . 

T ask 1 . Mutual understanding of project goals, reporting procedures, 
and liaison procedure between the subcontractor and the prime 
contractor were established. A PERT Chart, task chart, and work 
plan were developed. A list of otlier Great Cities and Cnllfornip 
dlstrictc were developed for site visits later in the project. 

This work resulted in a detailed project work plan document tiiat 
was sent to PGP for review. 

Task 2 . Analysis of instructional materials (IM) liandling practices 
in LAUSD, in two other Great Cities, and in selected Jalifornia 
districts was conducted. Also, an analysis of the interfaces that 
existed within LAUSD and between the District, County, and State 
agencies with respect to instructional materials was done. Data 
for these analyses were collected by questionnaire and interview. 

In addition, a literature survey and its assessment were performed. 
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Analysis of user needs w/is done 1 ti wliiili rlnsses ol ms<ts 
nnd tlielr needs wer<* i dent i f 1 ••d. D.itn wer<* (nllected by rjneslion- 
nalre and interview with adn 1 ii 1 s t rat 1 ve 1 librarians, tearliers, ami 
other relevrnt personnel in LAl'SD, in ntlier California Hrlw)ol 
districts, and in other (Ireat Cities. 

4. Cslii[; data from dasks ? ami J, tie* system requlreruMUs for 
I ii.strtjctlonal materials were determined In terms ot types, vnltines, 
and use patterns . Hie control , reporting: , arcotintinp, , and otiM’r 
nanngemcMit requirements that the system must meet v/ere hid lea ted. 
Tills work resulted in a Systeri Kenuirenents Document tliat was 
sent to r(;r for review. After review tlie document was finalised. 

d a s k 5 ■ Alternative system concepts wi*re e>iar.ined that met the system 
requirements developed in Task 4, to select the optimal concept or 
the best features of several concepts. A [reliminary system 
description document was prepared and sent to PGP for review. 

Task 6. System implementation tasks were definc^d and a tentative 
schedule developed. Also, prclinlnarv cost estimates for implcmcMi- 
tation were developed. A preliminary system implementation plai 
document was prepared and sent to PGP for re^'iew . 

VJork on Tasks 5 and 6 was done nearly sii ultancously . During the 
completing of these two tasks and starting Task 7, observation of this 
project was conducted by the interviewing team. 

rask 7 . Using the prclir.inary system description and tlic prelimi- 
nary system implementation plan developed in Tasks 5 and 6, along 
with PGP reviews of those two docurentr , a design review and 
concurrence package was to be made up. This package would con- 
sist primarily of the system description, bu^ without specific 
details applicable only to LAUSD. Other materials would include 
a checklist for use by the reviewers in evaluating the system 
design. Once prepared the package was to be distributed to the 
National Advisory Committee (their first work involvement in the 
project), the PGP, and others for review and evaluation. 

i*ask 8 . The reviews obtained from Task 7 are to be evaluated and 
incorporated in the system design where feasible. An informal 
report will be prepared and sent to PGP for review. 

Task 9 . Using the results of Tasks 3, 7, and 8 major subsystems 
of the overall system are to be defined and specifications for 
operations, software, and hardware arc to be developed for each 
subsystem. Also, specifications for qualification testing of 
the system and its subsystems, and for documentation necessary, 
arc to be developed. At this point briefings or information 
workshops will be conducted in other Great Cities and other 
school districtc. Once the final systems specifications document 
is prepared it will be delivered to the prime contractor, making 
up one-half of the final documentation for the project. 
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Tai^k._ IJ). From tlu> results nf T.isV. h ,ind the nwi'-v/s, n final 
tern Implenentatlon plan is to ho nroparinl that will inrlndf 
detailed schedule, procedures for ncra i tor i ne; and inprovinr. 
system performance, and cost data for each sten of inplement/itlon. 

Also, two monthly status report letters are Issued by the subcon- 
tractor project staff. One Is internal, going to the department manage i , 
and the other external, going to the Project Director In LAUSD. In 
addition, the subcontractor staff Issues detailed technical progress 
reports quarterly. A progress report was issued from the office of 
the Project Director In LAUSD on October 1, 1970 with the next report 
to be submitted January 1, 1971. 

PI ys teal /e nvironmental setti ng. The offices of both the prlnc con- 
trnctcc (I.AL'S’ 1 and the subcontractor (SDC) are located In the !,os An'V’les 
area. The work of tlie projet Is being done there. 

The contracting, managing, and coordinating activities of tin- project 
are being carried out in the l.lbr.iry Services Section of the Division of 
Special Services in the I.AUSD, a public school setting. The tcdinlcal 
support and production activities are being carried out in tlie Mbrary 
and Documentation System.s Department of tlie Public Systems Division of 
SDC, a private research and information technology company. 

Duration of this project was projected in the proposals as a 12 
month effort, starting .March 15, 1970 and ending March 16, 1971. How- 
ever, due to circumstances, the work starting date was delayed until 
June 1. With the termination date remaining the same, the early stages 
of the work program were under tine constraints. 





S V 



Chapter II: Parameters of the Project 



This chapter contains a description of project staffing and an 
Indexing of identified project outcomes. 

Project Structure 



Staff structure . P'lgure 3 re, resents the organizational structure 
vtithln this project. 

The LAUSD Project Director, while indicating 10% of time allocated 
on project, is in continuing liaison with subcontractors, particularly at 
the upper management levels, with the National Advisory Committee members, 
and with the Project Monitor and members of PGP. Wliile much of this liaison 
is concerned with topics other than Project AIMS, the frequency of this con- 
tact with the individuals concerned implies more actual time spent tlian the 
10% allocation. 

The National Advisory Committee overlaps in membership with a state 
Joint committee of Library and Audio-Visual Association personnel. This 
provides a secondary advisory and dissemination platform for the AIMS 
Project. 

The Deputy Dliei.ror (and Project Monitor), as the single LAUSD full- 
time professional staff member of the prime contractor, provides the con- 
tinuity in the various liaisons between all of the agencies, as well as 
the monitoring of all of the activities. For example, all contacts of the 
subcontractors with LAUSD personnel are arranged through the Project Monitor. 
All products from the subcontractor are routed through the Project Monitor 
to PGP. 

The PGP represents the interests ut various departments in the 
Special Services Division of LAUSD. They review and evaluate all the 
products of the subcontractor. Implementation of the system under design 
will directly affect their operations. 

In the subcontractor's office, the manager of the department in 
which the project work is being done monitors the progress anu nerve<' 
informally as a consultant to his Project Manager. The Assistant Project 
Manager is the only full-time project member on the subcontractor staff. 

All other members of the staff are specialists that work on the project 
as needed. 

Project roster . The following staff members (identified by Job 
title) were currently on the project and were interviewed for information 
about the AIMS Project, and about selected products of the project with 
which each was associated. This is not a complete list of staff. The 
products and management outcomes interviewed around are identified in 
the column to the right. Some products or outcomes were interviewed 
around more than once, in each case with a different interviewee. 
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*'/J J1L‘ P - t 1 I ” r _.Yi li 
' f’ p nnVi h 1 1- 
for till- Initl.itlon, (ominrt, 
■ind Mnnl prndiirf^ of the APIS 
proloct. At the tine of ohs.T 
vntlon FIi:^ <issl p,npcf. t to APIS 

W.T-. .10. 



I’rodlirts nnd ‘fniintp' m t 

Oiitroros Ifiterviowcd 

/ll'i’J'.'lLl JITJ- Jfi'irv I ••(./(><■ 

Stnf f - h) rioK, . 

Budget . 

I’ro )iTt Rpcc- i f i ent ions . 

•3t.Tf f - we 1 f nre . 

Adeqmte i itrr/intr.i- 

ngenry relationship... 
I’roject mnii.Tgrrent 

•decision Rtriicture. 



• H f 8 p o n s 1 b K* f o r t h e 
'•oordln.Ttlon of the efforts of 
tlie LAi:SD staff with the snb- 
•ontractor staff, and for monitoring 
product quality. Is cormltted 
to the APIS project throughout 
Its duration, with fTi: assignment 
of 1.00. 



Onallty control 

nechanlsm - schedu],.. 



j— L*inn. ^ ^ K fl C u 1 d .1 n c e l^ne 1 

Metier; Director of OatTTro- 
cesslng for l.AlhSD. Responsible 
for expressing needs of LAL'SI) 

■ind monitoring product quality 
and progress of project. Ulille 
not assigned directly to the 
project, he p.irt ic ipati's approxi- 
mately 107 . of his tine. 



i’roposal - LAUSI). 
.System requirements 
document . 



Subc ontractor Personnel ; 

.^broJecbJ^^J^; Responsible for 
all subcontractor staff efforts 
on the AIMS Project throughout its 
duration, with ITP assignment of ' 



Staff - welfare. 

Staff - hiring. 

Budget . 

Adequate Inter/lntra- 
agency relationships. 

Monthly reports - sub- 
contractor to prime 
contr ac tor . 

Interview. 

Detailed project plan. 
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FTE represents Full Time 
time th.it is alloc.ited 



Equivalency, the 
to this project. 



percent of the individual's 
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‘ 'i' ' Mf]i r.'H to*" Por^^onnol; t (.on t ’ niirr] ) 



Pr<o|iiri*^ anM Manor, *'n t 
nntcnn»'*> I n t o rv i r vod 
Around pr r I nj orv I ov/i*r 



- i 'd a n l 1 ' r oj r r t Mjajijaiif ' r i 
'O'.pMU'il’lr for as^i^tlnrt tin* 
i'ro)i*(t Manaror In managing the* 
*.uM< on f r or t o r ';laff of fort a and 
•or^inr on a'^sfgnod tasl'.s, Iq 
<o:;:ntforl tn tlw.' pffijoct tlirougli- 
r>u\ 1 f s dnraf ion , v/i th FI I 
as'i ir,nponl l.OM. 



^hioq 1 1 onna 1 ro . 

1 nt or V j ow , 

Monthlv ropnrt - •od)- 
coat rar tor to pr I '• 
ron t rar tor , 

Journal art Iclo . 



S o fd o r Sj^' s^t i ’j]s^ a Jj/ s t i 

j’ospt>n'- 1 b 1 1 1 1 V to project fi>r 
la'^ks in tlir dovolopnont of 
f inal **vslrr. roquironents docunont, 
i)ro 1 j nlnar> s**^ ton inpl cnen tat i on 
plan, and d«"nonf^tr at Ion modo] 
of tilt* system design. FTK assign- 
ri«*nt of .30 at the tine* of 
oh'u* rva 1 1 on . 



Systen rotnonts 

docunon t s , 

Pro 1 Inl nary system 
description. 

Pro 1 Inlnary Imp 1 enon ta t i on 
plan. 



Library Cost estimates. 

Systeins Spec 1 a^l 1 s t ; Rosponslblo 
to project for tasks in system 
fjiia 1 i f 1 cat Ion test design, costing, 
anti otlu‘r sy.>tem design tasks, 

KTI.' assignment of ,50 at time of 
observat ion , 



S enio r S yste ms Analyst : 
Responsible to project for tasks 
involving aud io -visual material 
system design, and other tasks 
wliere expertise Is relevant, 
ni.* assignment of ,15 at time of 
obsor nation. 



Audi o-vi sual system 
requi rements , 
Biblicgraphy , 
Interview, 



i'iie products and r.iinagement outcomes above were identified from various 
sources, e.g., proposals, project reports, interviews, etc. From a total 
oi 67 products and management outcomes Identified, the eight AIMS Project 
staff members were interviewed around 13 products and nine management outcom^is, 
or one-third of the 67 identified. 



The following two sections of this chapter provide indexes of pro- 
ducts and management outcomes, and interrelationship charts (product 
trc*e, managonont network) for the major products (or product groups) and 
major management outcomes. Chapters III and IV contain detailed descrip- 
tions the operations (taski^) to accomplish each analyzed product or 
outco«re, the enablers (knowledges, skills, sensitivities), and standards 
(criteria for adequacy). OQ'J 
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PrMflurts^ r <1 tj’d 

lable 1 1<, nn indoxir^; hv 1*‘V‘»1, nnd vni*‘^(^r\' 

wltti'iM l«>vc] wlit’rt' app] iral’li’, of all t!io 1 (l<-nt 1 f i cd nrodnrf, n1 l'r<’).(t 
It'*' incli’x also Indicator (lie procl'u tM tfiat arc sniMifir to proliTt 
nanap Tion t , a'> v<’ 1 1 as tlioRo around wliirii lntiTvif",.'s wore ( niuliu t<.' , 

r'o; ' Kip,tiro ^ in a schcnatfc of all t!io na):ir pTfsIm ts 
(l,fv«-ls I, il. III) Idontlfiod for this projoct. It attonpts f o slinv 
thf hiorarrlilral nr Intordopondoncy ro 1 at 1 onsh 1 ns of thi> products as 
thov servo the I (u a ] product (Level I), LooHmk at the finure fri'n 
liotlon to top represents tlio se',oence of najor products f ron tlie - tart 
of tile project to Its termination. The dotted line, sllphtlv ahove 
center, represents tl)o entrance of the 1 n tervl ewl nr, team Into this pro- 
)<-ct for observation and Inti-rvlew purposes. Prodin ts listed fielov the 
dott(d line were completed before the tli:ie of ohse rvat 1 r>n . 

An Important distinction Is necessary at tills point. If a product 
that Is specific to project management was Interviewed around from a 
ps-oductlon point of view, then that product will appear in Chapter III 
descriptions. If, however, that product w.is Interviewed .around fron a 
nanaKement point of view, it will appear In the Chapter IV descriptions. 



■Man agem ent Respon slb llltles (Out comes) 

Index of management responsibilities . Tal)le 2 Is an Indexlnp of 
all the Identified management responsibilities (or outcomes) of I’rojeet 
AIMS by level, and category within level where appllc.ible. The Index 
also indicates the management outcomes Interviewed arcund and contains 
products that are specific to management. 

Ma -icgement n e twor k. Figure 5 Is a schematic of all the major 
(Levels 1 and III) management outcomes identified for this project, 
showing their lilerardiical and interdependency relationships. Looking 
at tlie figure from bottom to top represents the sequence and interrela- 
tionships of the major management outcomes from initiation of the project 
to its termination. Outcomes around which Interviews were conducted 
are referenced by an arbitrary identification number. Their detailed 
descriptions ..lay be found in Chapter IV. 



TABLR 1 
Product Index 



LEVEL I: FOCAL PKODl'CT 

Design Gpecl f leal Icms and loplempMtat Um plan for an 
aulotoated Inat ruct ional^Baier lala^handl Ing syBlenii 



LEVEL II: TFAMINAI. FRODUCIS 

Final ayaleiD spec i f 1 cav Ions • 

Final ays ten inplenent at icn plan . 

Final aystcan requirements document. (P-45)^ 



LEVLl 111: INTEKMEDIATF FFOinJCTS 



I’rd liiiln.iry inpleracntat ion ayaten. (P-*06) 
Dealgn r«*vlew ^nd concurrence package. 
PeBlgn review informal report. 

Uner needs analysis documentation. 



Spe c H i ca t Ijon^ 

Systerm anMlyflls documentation. 
Preliminary system deacrlptlvn. 
(P-49) 

Syatem description* 

Preliminary system require- 
nrnts document . 



Pro cedures 
•Metalled 
project 
plan. 
(PM- 34) 



LEVIX IV; FOUKDATIONAI. PRODdCTS 



^cji^^d vj 1 V s 

Tentative implementation schedule. 

Dotnilod schedules for implementation plan. 
^Task cliart . 

♦PERT chart. 

tvaluatf'ut Instrume nts 
Pilot system. (l‘-48) 

Wo rkshop 

Infonaatlon workshop. 

Dat a C ol lection Instnimcnta 
Interview guide (LAUSD). 

Interview |ipilde (other cities). 

Questionnaire. (P-27) 

Check! 1st . 

Interview (protocol). (P-36) 

Procedures 

System qualification testing procedures. 
System Implementation organization and 
procedures . 

Syatem monitoring and improvement procedures. 
♦Project work plan. 

♦Project reporting procedurea. 

♦Coordination and liaison procedure® between 
contractor and subcontractor. 

Syatem design alternatives^ evaluation 
procedures . 



Reports 

Modified system dcnlgn. 

LAUSD, IM-hsndllng practices reports. 

Interface analysis report. 

IH-hsndllng practices In other cities 
reports . 

Journal article. (P-52) 

♦Monthly report - subcontractor to prime 
contractor. (D1-54) 

TWO other cities sltc-vlslt reports. 

LaUSD slte-vlalt reports. 

♦Monthly report - internal sub ontrsetor • 
Audio-visual system requirements. (P-71) 

Design review coomentSi. 

Bibliography. (P-38) 

Specif lest Iona 

Cost estimates for systiom. (P-03) 

Prlorlzed list of major subsystems. 

Hardware requirements. 

Software design specifications 
System docianentatlon specifications. 

Detailed task descriptions for Implemen- 
tation plan. 

Major cost categories for Implementation plan. 
List of user classes. 

List of «'ser needs. 

System design concepts. 

♦List of other cities and school distrlcta 
for visits. 

Proposal - prime contractor. (P-25) 

Proposal ** subcontractor. 



♦ Products around which Interviews were conducted. The number following the Item la an 
arbitrary product Identification number, and Is used in Identifying products for 
. description In Chapter III. 

° Asterisk Indicates products specific to project management. These products sppesr 
again in the management responsibility index. Table 2. 




Rpspons i b 1 1 I t V Indp/ 



UA'r.I. I: FMCAf- oi'TOiMl 

Sponr yt-iH cvptrd antornnt»'(l i ns t rtirl I mini — n t *’r 1 nl s-)inntl ling 
sysl»n design sf^'cH 1 ca 1 1 ons and If n 1 * -’en f at 1 on pi in. 



i.i;vr.i. Ill; 


IM T.PJIKDlATi. on (••VII S 


I’roduct 1 on Managi’mi-nt 


t]nvl ronno n t .1 1 Mnn.imnn’nt 



Manage production of docunu^nts. Staff - 1)1 ring. (l?t-57) 



Product spc’cl f 1 cations , O’M-59) 
Quality control mechnnlsn - f^chculul* 
(PM-h \) 

Staff - work s die du ling . (PM -6 7 ) 
*I)«‘tallcd prolect plnn.^ 


budget . ( i/1-S8) 

*s. Staff - welfare. (fri-bO) 

Adequate 1 nter / 1 ntra-agcncy re la- 
1 1 0 islil ns . (PM-Al ) 

Project nanag<*nent decision struc- 
ture. (i:M-h2) 

Maintained philosophy of approach 
to problt^n. 


u:vr:i. iv; 


KOL'NIJATION’M. OUTCOMFS 


IPJL nn^^eTTient 


Knvl ronnn*ntal Manngomi'nt 


*Tnsk chart. 

’^PI:KT chart. 

*^Work plan. 

^Llst of otlier cities and sciiool 
districts for visits. 


’^Project ronortlng procedures. 
*Coordi nation and liaison procedures 
betwei'n prime and subcontrac to r 
’^Monthl V report - subcontractor 
to prime contractor . 

*^Monthlv report - Internal sub- 
contractor . 


Management outcomes (and product^ 
which Interviews were conducted. 


• that ai e specific to management) around 
The number folloi^ing the item is an 



arbitrary Identification number, and Is used in idtMitifying outcomes and 
products for description in Chapter IV, 

Asterisk Indicates products spi'cific to project rnnagement (also appearing 
in Table 1). 
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June 1, 1970 



^ Indicates the product level (see Table 1) . 

^ Products around which interviews were conducted are followed by their identification 
number . 




FIG. 4. Ptoduct tree. 
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Indicates the management outcome level (see Table 2). 

^ Outcomes and responsibilities around which interviews were conducted are 
followed by their identification number. 

FIG • 5 « Management network • 
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Chapter III: Details on Each Product Development 



This chapter contains (^) a listing of all the standards, tasks, 
and enablers identified during interviews around each interviewed product; 
(b) summaries of citation within each category of standards, tasks, and 
enablers; and (c) the relationships of standard, task, and enabler cate- 
gories to product categories in terms of frequency. Details on management 
responsibilities will be detailed in the next chapter. 



Listing of Product Standards, Tasks, and Enablers 

Each product heading is followed by a brief annotation contaaiing: 

(a) whether the product was completed or uncompleted at time of observation 
and if completed, how recently; (^) a description of the product; (c) the 
level (foundational, intermediate, terminal, focal) of the product;”" and 
(d) the significant relationship (s) with other products of the project. 

Following the annotation, the work focus of the interviewee to the 
product will be identified, i.e., whether it was product production or 
production management. 

Then standards, tasks, and enablers appear under their separate 
subheadings to identify: 

1. The standards by which the product is judged or controlled. 

2. The tasks involved in generating the product. 

3. The enablers necessary for generating the product, stated in 
the form of (^) knowledges, (^) skills, and (O sensitivities 
to be possessed by project personnel. 



P-03: Cost Estimates for System 



This product had been completed a short time before the period of 
observation. It was a working report of the rough estimates of the cost 
of developing and installing each component subsystem, for planning, 
and for indicating the costs projected for the complete system. This 
is a Level IV (foundational) product. This product was developed as a 
part of the Preliminary System Implementation Plan (P-06) and was included 
in that document . 
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Work Focus of the Interviewee to This Product: 



This product was interviewed around once. The interviewee produced 
the product. 



Product Standards: 

1. Cost estimates provide accurate basis against which potential 

user of the system can make decisions about implementation 
and use • 

2. Cost estimates accurately reflect current costs of time, 

personnel , and equipment • 

3. How closely cost estimates for Implementation of automated 

information-processing system under design match with 
previous guesstimates by experts. 



Production Tasks: 

1. List components of automated information-processing system 

under design. 

2. Prioritize list of automated information-processing system 

components under design. 

3. Specify order of implementation of each component of the 

automated information-processing system, based on the 
prioritized list of components. 

4. Estimate amount of time needed to Implement each component 

of the system under design. 

5. Estimate manpower needs of implementation per system component. 

6. Estimate equipment needs of implementation per system component. 

7. Prepare written report of cost estimates for implementation 

of automated information-processing system. 

8. Estimate manpower costs per system component. 

9. Estimate equipment costs per system component. 



Enablers of Production: 

1. Knowledge of what parts of an information-processing system 

are done by clerical people. 

2. Knowledge of how data conversion is done in information 

processing. 

3. Knowledge of how much time is required to write and develop 

a computer program. 

4. Knowledge of personnel types required in a functioning 

automated information system. 

5. Ability to estimate automated information-processing system’s 

programming costs. 

6. Ability to observe a process and identify its component parts 

and the inputs and outputs of those parts. 
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P-06: Preliminary System Implementation Plan 



This product was completed just as observation began. It was an 
informal report of tasks necessary' to Implement the system, a tentative 
schedule, and the cost estimates. This is a Level III (intermediate) 
produce. It was produced by the subcontractor and sent to the prime 
contractor for review. After review, the product fed into the Design 
Review and Concurrence Package. 



Work Focus of the Interviewee to This Product: 

This product was interviewed arouid once. The interviewee helped 
(team effort) produce the product. 



Product Standards: 

1. Workability of the Implementation plan. 



Production Tasks: 

1. Utilize other previous project resources and documents. 

2. Exchange information with project staff to come to a decision 

for each requirement of the system. 

3. Provide an explanation of what a system implementation plan is. 

4. Formulate a set of questions that could be applied/asked 

relative to general considerations or functions of the system. 

5. Provide a description of the characteristics of the general 

^ considerations to be dealt with relative to specific problem 

areas or specific functions in the system. 



Enablers of Production: 

1. Know some of the procedures of non-math operations research. 

2. Know the kind of questions to ask relative to determining 

general considerations and specific functions of the plan. 

3. Know how to do a cost-benefit analysis. 

4. Know how to do a trade-off study. 

5. Skill in developing a system implementation plan — which comes 

from experience in preparing such plans » 

6. Skill in reducing data or information to a simple statement. 

7. Ability to make qualitative decisions. 

8. Ability to think logically — as to component elements within 

the implementation plan. 




P-25: 



Proposai-LAl’SD (Prime Contractor) 




i'h rough this product the project was successfully funded. It 
contained the specifications for the proposed stuc’y (the AIMS Project). 
This is a Level IV (foundational) product. 



Work Focus of the Interviewee to This Product: 

This product was interviewed around once. The interviewee helped 
produce the product. 



Product Standards : 

1. Satisfaction of funding agency. 

2. It makes good sense to me. (Relates to Tasks 2 and 5.) 



Production Tasks: 

1. Determine just what activities were taking pl£.ce with respect 

to the problem. 

2. Estimate the scope of change required with respect to existing 

system . 

3. Plan to modify existing system rather than imposing new 

structure on old. 

4. Exchange ideas with others in the planning group. 

5. Estimate if specifications are general and/or flexible enough 

for use in other cities. 



Enablers of Production : 

1. Knowledge of current methods of instructional-materials-handling 

operations . 

2. Knowledge of the design and service of existing materials 

handling systems. 

3. Knowledge of how instructional materials inventory system 

relates to the physical accounting system. 

4. Skill in identifying clear expression from a neutral, unbiased 

viewpoint . 

5. Skill in differentiating logical from illogical sequences. 

6. Sensitivity to need to solve problems rather than create system 

for system’s sake. 

7. Sensitivity to the basic problems of students and instructors 

with respect to material needs, n 
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P-27 : Questionnaire 



This product had been completed for some time at the time of 
observation. It was a form, used only in LAUSD, requesting information 
about instructional-materials-handling practices in c»ach of the instructional- 
materials agencies. It was mailed to the agency heads and the information 
was used preparatory to later site visits. This is a Level IV (foundational) 
product. It was initially developed in the Detailed Project Plan effort 
and fed into both User Needs Analysis Documentation and Systems Analysis 
Documentation . 



Work Focus of the Interviewee to This Product: 

This product was interviewed around once. The interviewee helped 
(group effort-staff meeting) produce the product. 



Product Standards: 

1. The questionnaire gets the information necessary to do the 

tasks specified in the subcontractr*r 's proposal. 

2. The questionnaire contained questions that asked for information 

that could be responded to without excessive delay. 

3. It did not include questions that would best be answered in interviews. 

4. Keep the questionnaire as brief and easy to fill out as 

information needs permit. 

5. It would produce the detail of information needed, e.g.. In 

terms of numbers and people. 



Production Tasks: 

1. Decide (staff, in group) what information was needed to design 

the system. 

2. Write (staff, in group) in outline, what information itemc> ^ 

needed to design the system. 

3. Refine (staff, in group) list of information items that was 

needed. 

4. Brainstorm (staff, in ^"group) all the details of the system 

design phase that would have to be accommodated. 

5. Review (staff, in group) the subcontractor's proposal to 

identify what questionnaire should cover. 

6. Decide (staff, in group) what information could be better 
gotten by questionnaire than interview. 

Decide (staff, in group) on degree of structuredness or open- 
endedness of questions to elicit information needed. 
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Knahlers c Production: 

1. Knowledge of wh it kinds of questions to ask to solve design 

needs, such a:-! size of effort, kinds of tasks, number of people. 

2. Knowledge of information processing, 

3. Able to prepare a questionnaire to elicit from the respondent 

the information required . 

4. Skill in working in a group — able to listen to other people, 

make some suggestions, then work to agreement. 

5. Able to design a question that is not too restrictive or too 

open-ended for the respondent, 

6. Sensitive to what other people have to say in a group session. 



P-34: Detailed Project Plan 



This product had been completed for some time at the time of obser- 
vation. It was an informal report from the subcontractor to the prime 
contractor for r view. It included a PERT chart, list of other cities 
and school districts for visits, project reporting procedures, co- 
ordination and liaison procedures between prime and subcontractors, 
task chart, interview guide, a’^d questionnaire. This is a Level III 
(intermediate) product. This product was developed from reviews of the 
proposals (prime and subcontract), and from meetings between the two 
agencies. After the document was reviewed and formalized it provided 
the framework in which the project work was carried out and fed directly 
into the Systems Analysis Documentation and the User Needs Analysis 
Documentation , 



Work Focus of the Interviewee to This Product: 

This product was interviewed around once. The interviewee pro- 
duced the product. 



Product Standards : 

No information collected under the heading. 
Production Tasks: 

1. Review proposal to break out work units. 
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2. Allocate/derormine manpower needed to carry out tasks. 

3. Determine a work schedule for the project. 

A. Divide work up into tasks relative to the project. 

5. Determine the nature of products that are produced. 



Enablers of Production: 

1. Know the capabilities of your staff. 

2. Ability to estimate time relative to a particular task-- 

based on experience. 

3. Awareness of the magnitude of the work on the project. 



P-36: Interview (Protocol) 



The interviews in all settings (LAUSD, other California city 
school districts, other Great Cities school districts) had been com- 
pleted for some time at the time of observation. The interviews were 
conducted to collect data on instructional-materials systems in all 
the settings cited above. Where appropriate an interview guide was 
used. This is a Level IV (foundational) product. This product produced 
data essential to the Systems Analysis Documentation and somewhat essen- 
tial to the User Needs Analysis Documentation. 



Work Focus of the Interviewees to This Product: 

The product was interviewed around three times. The interviewees 
helped (team effort) produce the product. 



Product Standards : 

1. An interview was terminated when it stopped producing any 

new information as judged by the interviewer . 

2. An interview guide did assure that the interviews produced 

at least a minimum amount of information. 

3. Points in the outline or interview guide have been covered. 

4. System analysis interview provides information about what 

is input and output, form of input and output, actual fre- 
quency of output, desired frequency of output, channels 
through which information passes. 
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1. Conduct forum 1 interviews with metropolitan s('lu)ol dLstrict 

pc‘rsonnel ns i np, an Interview guide. 

2 . (-onduct informal hut focused intei*vicws (e.g., over lunch 

or drink), probing ins truct i onal -mate rials-hnnd 1 ing practices. 
Write inhouse memo for project staff describing problems and 
processes (^f instructionaJ-materinJs-hnndllng in metropoli- 
tan school districts relating them to local effort. 

4. Reference interview data against needs stated in proposal. 

5. Question sciiool personnel using an interview guide. 

(). Make notes during interview of relevant information. 

7. Write report (inhouse memo) and distribute to other pro- 

ject staff. 

8. Agree (project staff) before doing interviews to a set of 

.data tlie interviews should produce. 

9. Ask questions relative to the operation of an ins tuctionai- 

mate rials cente r and/or an audio-visual center . 

10. Take notes during the interview. 

11. Tape record tlie interview. 

12. Acquire any products tiie agency is able to give away. 

13. Digest the information to translate it into the System Re- 

qui remen ts Document . 



Enablers of Production: 

1. Know what a "typical" school district operation is, relative 

to instructional materials. 

2. Knowledge of instructional materials used in schools. 

3. Knowledge of library practices in handling instructional 

mate rials . 

4. Able to spot differences of operation and organizational 

structure among school districts relative to the handling 
of instructional materials . 

5. Able to extract information one is after from an interviewee. 

6. Able to conduct effective interview bv keeping interviewee 

focused on subject interviewer is interested in. 

7. Able to gather information in an informal interview over a 

lunch or drink. 

8. Able to extract from people what you need to know, keep 

inte rview focused . 

9. Able to establish some rapport with the interviewee. 

10. Sensitivity to the fact that some questions may be mean- 

ingless or not polite to ask, based on responses to other 
questions . 

11. Sensitive to keeping an interview focused without antagonizing 
interviewee . 

An interest in and tolerance for people and tlieir problems. 
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: lUbl logrnphy 



Tills product Had been completed for some time at tlie tine of 
observation. It was an iniiouse , working bibliography from a survey 
of the information-science literature for the past five years (1965-70), 
looking for new and innovative approaches to the whole area of in- 
formation handling systems at the noncollege level. This is a Level 
IV (foundational) product. This product was developed in tiie systems 
analysis effort and included in tiic Systems Analysis Documentation, 

It fed into both Preliminary and Final System Requi re»7ients Documents 
(P-45) and tlien into the Preliminary System Description (P-49). 



Work Focus of the Interviewee to This Product: 

This product was interviewed around one time. The interviewee 
produced the product . 



Product Standards : 

1. The bibliography represented an estimated 60 to 70 per- 

cent of the relevant information-science literature pub- 
lished from 1965 to 1970. 

2. Cost-benefit ratios in producing the bibliography are maximized 

in terms of time spent. 



Production Tasks: 

1. Scan indexes and guides to periodic information-science liter- 

ature in library. 

2. Identify titles of materials and articles that may have rele- 

vant information to area of information handling systems 
at the noncollege level , 

3. Locate articles and materials of interest on library shelves. 

4. Skim articles and materials for relevance of content to 

information need and salient details. 

5. Talk with experts in information science to locate addition- 

al sources of relevant information. 

6. Write on cards salient details from each of the relevant 

articles or materials located , 



Enablers of Production: 

1. Knowledge of information science and, in this case, knowledge 
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o r au(] i o-v I sua I d I sscm 1 nat I on sys tor is . 

2, Ahlf* to do bajilc 1 1 b rary / P terature r«»srarrh tasks, i.o., 
using 1 lt<*raturi* indoxos and guides . 
i. Ai)Ie to use tine In library effectively with high Information 
output pe r given amount of tine spent . 



P-A5: Final System Requirements Document 



Tliis product had been completed for some tine at the time of 
observation. It was a formal report of the requirements the system 
under design would have to meet in terms of instructional-materials 
type volumes and use patterns. Included were tlie control, reporting, 
accounting, and other management requirements the system would liave to 
accommodate. This is a Level II (terminal) product. This product was 
the finalized version of the requirements based on tl^e Preliminary Sys- 
tem Requirements Document and the LAUSD review of that document submitted 
by SDC. This product provided the basis for the development of both the 
Preliminary System Description (P-A9) and the Preliminary System Imple- 
mentation Plan (P-06). 



Work Focus of the Interviewee to This Product: 

This product was interviewed around once. The interviewee helped 
(team effort) produce the product and also managed the production. 

Tlie focus here is both product production and production management. 




Product Standards: 



1 . 



2 . 



3. 

4 . 

5. 

6 . 



7. 



Parallelism in terms of the organized structure of content 
within the document . 

Degree of specif icity — was it too /not specific enough to 
be effective? 

Lack of inconsistencies in the content. 

Positive feedback from uninvolved party, i.e., not an author. 

Positive feedback/reactions from recipient/user agency. 

Time — in terms of the bands of time establislied in conjunc- 
tion to. the ’’clip-level’’ — acceptable quality within time 
frame . 

Type print — determine whether or not there are light hits 
(type) that will fade out. 

Type will reproduce clearly on final copy. 
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9. Correction marks will not reproduce on final copy. 

10. There is continuity and flow in terms of the language. 

11. There are reader guidelines or page headings on each page. 

12. Positive feedback from reviews by people internal and ex- 

ternal to the project. 

13. Compare, by reading, what you want to say to what you have 

written. 

14. A feeling that says : am comfortable in letting somecjne 

else review/read what I have written.'* 

15. Could no longer think of content or format that should be 

expanded or changed. 



Production Tasks: 

1. Review survey reports and previously written project 

materials. 

2. Talk to other staff members for input information. 

3. Establish a format for the document. 

4. Provide a description for each subparagraph as to content. 

5. Establish a "clip-level** (tolerance level) for exactness 

of language, completeness of material in terms of time 
and resources available. 

6. Set up mechanical (typing, spacing) requirements with the 

typing pool. 

7. Determine the system requirements in terms of format and 

technical logic so that each section is consistent with 
other sections of the document. 

8. Prepare a table of contents for the document. 

9. Prepare an outline of the document. 

10. Insure language level is suitable for the reader/user — 

minimum jargon. 

11. Proofread the copy — in terms of content and mechanics. 

12. Evaluate or verify system limits (quantity/ jnput) as to 

reasonableness , meaningfulness , consistency. 

13. Review document for accuracy, completeness, internal con- 

sistency. 

14. Modify or add to rough draft sections of the document. 



Production Management Tasks: 

1. Work out a production schedule. 

2 . Insure/monitor the production schedule . 

3. Arrange to get report cover reproduced. 

4. Arrange to have document reviewed by the Division Editor. 

5. Arrange to have document duplicated. 

6. Needle writers to insure sections of the document are writ- 

ten on time. 



29 



4-08 



870 



KnablerH of Troductlon and Production Management: 



1. Knowledge of what technical writing/communication Is. 

2. Knowledge of coherence, unity, clarity rules for writing. 

3. Knowledge of the required level of detail In the document. 

6. Know the requirements of typography. 

5. Know how the agency typists work. 

6. Know the subdivisions of the document so can break out pro- 

duction chunks. 

7. Know the length of time It takes to type the document. 

8. Know how long It takes to proofread and make corrections. 

9. Know the length of time it takes to reproduce needed copies 

o. the document. 

10. Know what kinds of resources are available — typists. 

11. Knew something about the temperament of the support people. 

12. Know the production status of each piece of the document. 

13. Knowledge of how systems, in general, work. 

14. Know the purpose of the document. 

15. Know the concept of what is to result or evolve from the 

document . 

16. Ability to shift points of view, i.e., author to reader. 

17. Understanding of what the author is trying to accomplish 

or say so you won't change his material because you might 
have said it differently. 

18. Application of rules of coherence, unity, clarity. 

19. Ability to recognize/judge a reasonable amount of coverage 

to the material. 

20. Ability to quickly extract essential information from pre- 

viously written materials. 

21. Ability to express or transfer thoughts in writing. 

22. Ability to think logically. 

23. Aware of the corporate Image or style of writing or pub- 

lication. 



P-48: Pilot System 



This product was completed just prior to the time of observation. 
However, some additional work was being done during the observation 
time to prepare the pilot system for a demonstration. The pilot system 
is a file management system used as a simulation of the system under 
design for testing and developing operational standards. This is a 
Level IV (foundational) product. The pilot system was developed along 
with the designing of the system. It feeds into the Prel'^ Inary Sys- 
tem Description (P-49) primarily, but also feeds into the i^reliminary 
System Implementation Plan (P-06). . 
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Work Focus of the Intt'rviewej to This Product: 

'rhis product was interviewed around once. The interviewee pro- 
duced the product. 



Produc t Standards : 

No information collected und^ r this heading. 



Production Tasks : 

1. F<eview ^Requirements Document to determine hov; closely 

it could be simulated (information blocks and combinations 
thereof) . 

2. Read mauerials relative to file management system DS2. 

3. Talk to technical people responsible for file management 

system to gather information about the system. 

4. Analyze catalogs and available materials relative to school 

district instructional materials . 

5. Establish categories of information to go into the model 

system . 

6. Determine if categories of information can be recreated 

and combined. 

7. Analyze existing format and elements in the catalog to de- 

termine different kinds of instructional materials within 
a district. 

8. Determine process requirements of a file management system. 

9 . Determine minimum amount of data required to test the sys- 

tem. 

10. Work backwards from a successful run of the data and list 
where to put data from the punched cards . 



Enablers of Production: 

1. Know the System Requirements Document — its content. 

2. Know the purpose of the demonstration model. 

3. Know the output requirements . 

4. Know information retrieval systems — capabilities and how some 

of them work. 

5. Know what a file management system is — capabilities and how 

it works. 

6. Know about data entry — how it works. 

7. Know how fixed-format blocked records are utilized. 

8. Ability to design report formats — being able to extract in- 

formation contained within a simulation model and trans- 
late into a meaningful package format. 

9. Abii,* ty to design a file management system. 
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10. Abillt.y to 

of a lot 

11. Ability to 

12. Ability to 

sti*ucture 

13. Willingness 



design a computer based system with a 
of information and output components, 
interpret specifications . 
t ranslate/ u»ange existing :^nformation 

• 

to experiment and hypothesize. 



capability 



into another 



P-49: Preliminary System Description 



This product was completed just prior to the time of observation. 
It was an informal report to the prime contractor (from the sub- 
contractor) of the system design, including functional flow diagrams , 
hardware, software, procedural requirements, input/output (l/O) devices 
and file structures, requirements for personnel and their training, 
and system qualification testing procedures. This is a Level III 
(intermediate) product. The primary input to this product was from the 
Final System Requirements Document (P-45). After review by the prime 
contractor, the product feeds into the Design Review and Concurrence 
Package making up most of that document. 



Work Focus of the Interviewee to This Product: 

This product was interviewed around once. The interviewee helped 
(team effort) produce the product and also helped (team effort) 
manage the production of the product. The focus here is both 
product production and production management. 



Product Standards: 

1. Use of flow charts reveals consistencies in logic of the 

description . 

2. Everything has been included; it is complete in terms of 

what is written. 

3. Material actually says what I think it should. 

4. There are internal consistencies/logic within the document. 

5. Compared with System Requirements Document all the requirements 
have been covered and haven't introducad something that j.sn't 
required. 

Positive feedback from the project staff. 
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Production Tasks: 

1. Exchange information and interact with project staff. 

2. Read basic project materials — Interview reports, summaries 

of meetings, status reports, documents — to gather 1 lEorma- 
tion. 

3. Develop alternati. ves in terms of the contents of files. 

A. Describe how the design of the system is to work. 

5. Make a list of items of interest to a particular function 

relative to establishing contents of files. 

6. Write materials, set aside, come back and v;ork on them again 

(called germination) . 

7. Read through materials quickly for a gross understanding. 

8 . Reread in detail to make notations and corrections . 

9. If a point is misunderstood, ask the author for clarification. 

10. Critique writing to ensure consistency in point of view/ 

writing style in the document . 



Production Management Tasks: 

1. Establish a production schedule. 

2. Allocate work units/tasks. 

3. Monitor the progress of document production. 

A, Provide ‘^stimuli” to see that writers and support staff com- 
pleted their tasks on time . 



Enablers of Production and Production Management: 

1. Know what a system description does — its purpose. 

2. Know what a system design is. 

3. Understand what input/output requirements are. 

4. Knowledge of data processing techniques. 

5 . Knowledge of data preparation techniques/methods . 

6. Know the purpose of the document-situation specific. 

7. Ability to comprehend the logic expounded in the components 

of the documents. 

8. Ability to comprehend the utility of the materials in the 

document . 

9. Ability to detect gaps or vacancies in the document and its 

components . 

10. Awareness of the techniques, technology, and development in 
methodology that will impact upon your system within the 
next 5-10 years — don*t build an obsolete system. 
Understand the flow or working of the system itself. 
Sensitivity to the fear of change that exists in people. 
Awareness that one cannot dictate to people. 
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Journal Article? 



? 

» At tlie time of observation, this product was not completed, but work 

on it was to begin soon. The article, when completed, was intended for 
^ a professional library journal and would describe the system under design 

and its purposes. This is a Level IV (foundational) product. This 
product is urimarily a limited-audience information dissemination effort. 
In content it will rely heavily on the Preliminary System Description 
document (P-A9). 



Work Focus of the Interviewee to This Product: 

This product v;as interviewed around once. The interviewee would 
produce the product. 



Product Standards: 

1. Article length within limits estimated from journal review. 

2. Level of detail (technical level) not beyond assessment of 

audience skill level and interest. 

3. Other project staff members judgm'^nt of accuracy of description 

of ALMS Project and its goals. 



Production Tasks: 



1. Review journal for which article intended. 

2. Assess who is audience from journal review. 

3. Outline the article into its key points. 

A. Decide article length from journal review. 

5. Decide technical level from assessment of audiences (skill 

level and interest). 

6. Write rough draft using typewriter. 

7. liave other staff members review and edit rough draft. 

8. Write final draft using staff reviews. 



Enablers of Production: 



1. Thorough knowledge of English and English grammar. 

2. Some knowledge of information processing in lil)rary systems. 

3. Able to use the language to communicate clearly . 

A. Able to put yourself in the place of the reader by your 

imagination. 
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5. Sensitive to what you anticipate the audience will be inter- 
ested in knowing, their problems, and tlieir skill level in 
using automated inf ormation-liandling systems. 



PM-54: Monthly Report (Subcontractor to Prime Contractor) 



This product is a technical progress report issued monthly to the 
prime contractor. It is a summary of the major activities in the sub- 
contractor’s offices covering from the 15th to the 15tli of eacli month. 
Tl^is is a Level IV (foundational) product. This is an information- 
giving product for management and coordination purposes and not related 
to other products of tlie project in a production sense. 

Work focus of the Interviewees to This Product: 

Tliis product was interviewed around twice. One interviewee produced 
the product. Another was concerned with production management. 



Product Standards : 

1. Report doesn’t exceed a length of one and one-half pages. 

2. Report covers, in summary^, the significant activities of the 



in that staff. 

3. Positive feedback from the customer relative to the report 

content . 

4. Positive feedback from the department manager. 

5. No misinterpretations in the report noted in customer feedback 

6. Sense of satis faction tliat report is complete . 

7. All material that is important or necessary has been included 

in the report. 

8. Problems have been resolved — in relation to the content of 

report . 

9. Material in document is interrelated and fits together. 



Production Tasks : 



subcontractor staff as confirmed by the Project Manager 



Review what subcon' ractor staff has done or written during time 
since last report . 





1 . 



2. Hoview work schedule for project. 

IU‘view last rc'port to determine* what that rc^port had covorc*<L 
^ 4 . hr(*parc outline of major accompli sliments and activities, 

including important meetings that occurred in relevant time 
period . 

5* Write the rough draft of the report using a tynewriter. 

6. Have subcontractor Project Manager edit tlie report rougli 

draft . 

7. Using editorial cormients, finalise the report and have retyped 

by secretary. 

8. Send finalized copy of report to prime contractor Project 

Monitor . 

9. Use an earlier report as a guide. 



Production Management Tasks: 

1. Isolate main efforts of report period. 

2. Describe main efforts of report period. 

3. Insure coverage in report in terms of what is outlined in 

proposal . 

4. See that mechanical/production steps to produce the report 

copies are carried out in time . 

5. Insure that writer gets input information (primarily from 

the project staff) for the report. 

6 . Reread proposal to insure necessary points have been covered 

in the report. 

7 . Read entire report to insure completeness . 




Enablers of Production and Production Management: 



1 . 

2 . 

3. 

4. 

5. 



6 . 

7. 

8 . 



Some knowledge of information retrieval in computerized 
information-handling systems. 

Able to identify the major activities of the reporting period. 

Able to summarize the activities in a concise written form. 

Able to decide the amount of detail reported in terms of 
the limited lengths specified for the report. 

Technical knowledge of content which is the substance of the 
report . 

Ability to construct an informative and succinct report. 

Technical skill to spot things that are not x^ell explicated. 

Able to keep total picture of project in order to fit together 
the pieces of the report . 

Able to describe/divide system in a logical manner. 






9 . 
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P-71: Audio-Visual Instructional Materials System Requirements 

1 



This product had been completed for some time at the time of 
observation. It was an inhouse, working report of the special constraints 
unique to films and other nonbook materials that the system design would 
have to accommodate. It also listed ways the system could be designed 
to meet those constraints. This is a Level IV (foundational) product. 

This product developed out of the Systems Analysis Documentation and 
the User Needs Analysis Documentation and fed into both Preliminary and 
Final System Requirements Documents (P-45) . 



Work Focus of the Interviewee to This Product: 

This product was interviewed around once. The interviewee produced 
the product. 



Product Standards: 

1. The requirements are acceptable when the components and functions 

of an audio-visual system are clearly specified. 

2. The requirements must state the maximum flow, in terms of 

volume and frequency, the system can handle. 

3. The requirements for the system as stated in the report meet 

the standards (philosophy) of the Project Manager. 



Production Tasks: 

1. Identify the essential components for a functioning audio- 

visual (A-V) materials dissemination system. 

2. List ways components of a functional A-V materials dissemination 

system can be designed. 

3. Write design requirements of a proposed system to allow max,*:..am 

information flow in terms of volume and frequency. 

4. Draw a flow-chart of the proposed system. 

5. Specify what happens to each form of output — where it goes 

next in the system. 

6. Interface with other project staff to synthesize ideas on 

system design. 



Enablers of Production: 



1 . 



Know the booking cycle time requirements for automated A-V 
materials dissemination systems « 

Able to develop outline of items to cover in writing a report. 



37 . 

- 
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3. Able to do a PERT analysis of a process. 

4. Able to use flowcharts to systematically chart a process in 

terms of cell and inputs and outputs of those cells. 



Summaries of Product Data 



Included here are summary tables that provide the frequency with which 
interviewees cited specific standards, tasks, and enablers within specified 
categories that were arbitrarily defined for this study. 

Table 3 presents the categories of product standards and the frequency 
of citation within each. The categories cited most often, i.e., ’’complete- 
ness of content,” ’’communication and clarity,” reflect the developmental 
nature of the project. The system design is in the documents of the project. 
The documents, to be adequate, should be comple'.e and clear. 

Table 4 presents the categories of production tasks and the frequency 
of citation within each. Of the total number of tasks (113) cited, 32 were 
tasks of designing the output, 26 were tasks of clarifying the problem, 
and 21 tasks involved assessing the quality of generated outputs. No pro- 
duction tasks were cited that have to do with diffusing the output. This 
reflects the current efforts of the project in terms of designing and 
developing the system. 

Table 5 presents the arbitrarily established categories of knowledges 
and the frequency of citation within each. Most of the knowledges cited 
tend to occur in the categories of ’’technical topics” and ’’resources: 
personnel,” reflecting the reliance of the project upon technical expertise 
in systems analysis and design from a production point of view. From a 
management point of view it reflects a need for knowledge of present 
staff capabilities and commitments, as well as where to look for people 
that meet the requirements of the project. 

Table 6 presents the categories of skills and the frequency of cita- 
tion within each. ’’Finding fits/integrating” and ’’Planning/conceptualizing” 
are the two categories most frequently cited. Again these reflect the na- 
ture of the project, with the most relevant skills being production-related 
abilities in logical, sequential organization of information. 

Table 7 presents the categories of sensitivities and the frequency of 
citation within each. In this project great emphasis is placed upon aware- 
ness of capabilities, constraints, and problems of staff members, and upon 
potential users of the system under design. Especially important from a 
management point of view was the facilitation of the work relationship 
^ Lween the prime contractor and the subcontractor. 
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TABLE 3 

Frequencies of Citation of Product 
Standards in Each Product Standards Category 



Categories of Product Standards 

Completeness of content 
Quantity of outputs/data 
Quantity of effort expended 
Communication and clarity 
Utility or value 

Acceptancy by users 
Personal satlsfactlon/feellng 
Agreement/concurrence w/others 
Lack of errors/discrepancies 
Obvious (direc .) termination 

Appropriate design/content 
Goal attainment 

Acceptance by others (in project) 
Acceptance by sponsor 
Compliance w/sponsor guideline 

Compares favorably 
Internally consistent 
Satisfactory appearance 
Logical criteria 
Performs consistently 

Sources of variance controlled 
Functions as planned 
Successfully constralns/guldes 



Frequency 
of Citation 

12 

0 

0 

8 

5 

2 

7 

7 

5 
0 

1 

3 

4 
2 
2 

3 

6 
3 
1 
0 

0 

0 

0 



Total Number of Standards Cited 



71 
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TABLE 4 

Frequencies of Citation of Production 
Tasks in Each Production Tasks Category 



Frequency 

Categories of Production Tasks of Citation 

Clarifying problem addressed 26 

Formulating objectives S 

Designing the output 32 

Producing the output 16 

Collecting/ processing data 10 

Assessing the output quality 21 

Diffusing the output 0 

Total Number of Tasks Cited 113 




40 
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TABLE 5 

Frequencies of Citation of Knowledges 
in Each Knowledges Category 



Frequency 

Categories of Knowledges of Citations 

Standard school subjects 2 

Subjects learned courses 0 

Subjects related to RDD&E 3 

Technical professional topics 29 

Project focus topics, external 3 

Proj ect variables : external 6 

Project operation: general 6 

Project operation: specific 8 

Scheduling and organizing 8 

Staff status/ responsibilities 2 

Fiscal matters 0 

Resources : personnel 11 

Resources: money 0 

Resources: time 0 

Resources: equipment 0 

Guidelines for reporting 1 

Writing styles 1 

Staff competencies/ interests 0 

Technical terminology 0 

Sponsor concerns 0 

Management techniques 1 



Total Number of Knowledges Cited 




A1 



\/ 
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TABLE 6 

Frequencies of Citation of Skills 
in Each Skills Category 



Categories of Skills 


Frequency 
of Citation 


Teaching 


0 


facilitating people interactions 


3 


Translating content to media 


0 


Using/applying feedback 


0 


Programming project events 


4 


Programming subject matter 


0 


Programming technical equipment 


0 


Analytical reading/ study 


3 


Analytical problem solving 


4 


Analytical data handling 


2 


Disciplining self 


4 


Disciplining others 


0 


Listening 


1 


Writing 


6 


Presenting orally 


2 


Using media 


0 


Interpreting language 


3 


Finaing fits/integrating 


12 


Planning/ conceptualizing 


9 


Exercising judgment 


4 


Tracking activities/goals 


1 


Estimating expenses /resources 


3 


Persuading/ jus t if ying 


1 


Explicating goals/ procedures 


1 


Applying measurement tools 


4 


Locating/maintaining information 


1 


Using equipment/systems 


0 


Running task oriented meetings 


0 


Getting others to perform 


0 


Adapting to situation/demands 


0 


Taking another perspective 


2 


Total Number of Skills Cited 


70 
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TABLE 7 

Frequencies of Citation of Sensitivities 
in Each Sensitivities Category 



Frequency 

Categories of Sensitivities of Citation 



Values of self and others 3 
Capabilities and limitations 10 
Needs of self and others 3 
Interactions of self and others 2 
Context of subject matter 0 

Worth in disciplines /methods 0 
Context of objectives 1 
Worth in objectives 0 
Awareness of alternatives 1 
Awareness of structure 1 

Awareness of method 0 
Role of catalyst/synthesizer 0 
Language barriers 0 
Reality in goal setting 2 
Degrees of freedom to deviate 1 

Ex* sting value systems 1 
Personality of others 1 
Potential conflicts of interest 1 
Supportiveness required 0 
Unstated obligations 0 

Limitations of analysis /data 0 
Responses of target audience 2 
Cost/ benefit factors 0 
Sources of error 1 
Individual differences 0 

Recognition of data needed 0 
Acceptability of output 1 
Admitting error/adapting 1 
Willingness to experiment 1 



Total Number of Sensitivities Cited 




^ 3 , 




33 
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iliis scrtion provkicH siimnary Tables H throiipli 1!’ tliat present 
tile relationship of standard, task, and enabler categories by fr*'f|nenrv 
of occurrence to the prodvict focus categories of research, development, 
difliisloM, and evaluation products. Tlie reader will note tb.it in eacli 
talile the proiKuct focus category of "development" contains the liigliest 
frequency, followed hy tlie "diffusion" category. The other two categories 
In each case contain relatively small frequencies. The liigh frequencies 
of standards, tasks, and enablers in tlie "development" category reflect 
tile nature of tlie products of the project and the project itself. The 
project is considered a "development" effort. 

The liigh frequencies of descriptors in the "diffusion" category 
refkcts another Important aspect of tills project. There is a great 
deal of di '^fusion in this project, but in a special sense. Since the 
subcontractor is producing most of tlie products of this project, due 
to the highlv technical nature of tlie work, all the products (documents' 
are sent to the prime contractor's office for review. This is one level 
of diffusion within the project; however, there are two more that are 
Ii’iportant--one still within the project. The second level of diffusion 
within the project is to the National Advisory Committee. There are 
also diffusion efforts by means of a Journal article and of information 
workshops. The latter diffusion efforts will not occur until the final 
stage of the project. 




44 /f'O 
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TABLE 8 

Relation of Product Standards 
to Product Categories 



Categories of Product Standards 


Product Focus 


Research 


Development 


Diffusion 


Evaluation 


Completeness of content 


3 


3 


5 


1 


Quantity of outputs /data 


0 


0 


0 


0 


Quantity of effort expended 


0 


0 


0 


0 


Communication and clarity 


0 


4 


4 


0 


Utility or value 


2 


1 


2 


0 


Acceptance by users 


0 


1 


1 


0 


Personal satisfaction/feeling 


0 


6 


1 


0 


Agreement/concurrence with others 


0 


6 


1 


0 


Lack of errors /discrepancies 


0 


2 


3 


0 


Obvious (direct) termination 


0 


0 


0 


0 


Appropriateness of design 


1 


0 


0 


0 


Goal attainment 


0 


1 


2 


0 


Acceptance by others (in proj) 


0 


2 


2 


0 


Acceptance by sponsor 


0 


2 


0 


0 


Compliance with sponsor guide- 










lines 


0 


0 


2 


0 


Comp ares f a vo r ably 


0 


2 


1 


0 


Internally consistent 


0 


2 


4 


0 


Satisfactory appearance 


0 


3 


0 


0 


Logical criteria 


0 


0 


1 


0 


Performs consistently 


0 


0 


0 


0 


Sources of variance controlled 


0 


0 


0 


0 


Functions as planned 


0 


0 


0 


0 


Successfully cons trains /guides 


0 


0 


0 


0 


Total Standards per 
Product Focus 


6 


35 


29 


1 
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TABLE 9 

Relation of Production Tasks 
to Product Categories 



CateRories of Production Tasks 


Product Focus 


Research 


Development 


Diffusion 


Evaluation 


Clarifying problem addressed 


2 


13 


7 


4 


Formulating objectives 


3 


5 


0 


0 


Designing the output 


3 


15 


9 


5 


Producing' the ou tpu t 


0 


3 


12 


1 


Collecting/processing data 


2 


1 


0 


7 


Assessing the output quality 


1 


5 


15 


0 


Diffusing the output 


0 


0 


0 


0 


Total Tasks per Product Focus 


11 


42 


43 


17 
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TABLE 10 

Relation of Knowledges to Product Categories 





Product Focus 


Categories of Knowledges 


Research 


Development 


Diffusion 


Evaluation 


Standard school subjects 


0 


1 


1 


0 


Subjects learned courses 


0 


0 


0 


0 


Subjects related to RDD&E 


0 


3 


0 


0 


Technical/ prof essional topics 


1 


14 


10 


4 


Project focus topics, external 


1 


1 


0 


1 


Pro j ec t variables : external 


1 


4 


1 


0 


Project operation: general 


0 


4 


1 


1 


Project operation: specific 


1 


4 


1 


2 


Scheduling 


0 


8 


0 


0 


Staff status/responsibilities 


0 


2 


0 


0 


Fiscal matters 


0 


0 


0 


0 


Resources : personnel 


0 


10 


1 


0 


Resources : money 


0 


0 


0 


0 


Resources : time 


0 


0 ■ 


0 


0 


Resources ; equipment 


0 


0 


0 


0 


Guidelines for reporting 


0 


1 


0 


0 


Writing styles 


0 


1 


0 


0 


Staff competencies/ interests 


0 


0 


0 


0 


Technical terminology /language 


0 


0 


0 


0 


Sponsor concerns 


0 


0 


0 


0 


Managemen t t echnlqu e s 


0 


1 


0 


0 


Total Knowledges per Product 
Focus 


4 


54 


15 


8 
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TABLE 11 



Relation of Skills to Product Categories 



Catej^ories of Skills 


Rroduct Focus 


Research 


Development 


Diffusion 


Evaluation 


Teaching 


0 


0 


0 


0 


Facilitating of people interactions 


. 2 


1 


0 


0 


Translating content to media 


0 


0 


0 


0 


Using/applying feedback 


0 


0 


0 


0 


Programming project/events 


0 


3 


1 


0 


Programming subject matter 


0 


0 


0 


0 


Programming technical equipment 


0 


0 


0 


0 


Analytical reading/study 


0 


2 


1 


0 


Analytical problem solving 


0 


2 


1 


1 


Analytical data handling 


0 


1 


1 


0 


Disciplining self 


0 


2 


2 


0 


Disciplining others 


0 


0 


0 


0 


Listening 


0 


1 


0 


0 


Wri ting 


0 


3 


3 


0 


Presenting orally 


0 


2 


0 


0 


Using media 


0 


0 


0 


0 


Interpreting language 


0 


2 


0 


1 


Finding fits/integrating 


0 


6 


5 


1 


Planning/concep tualizing 


2 


2 


2 


3 


Exercising judgment 


0 


3 


1 


0 


Tracking activities/goals 


0 


1 


0 


0 


Estimating expenses/resources 


0 


3 


0 


0 


Persuading/ jus tifying 


0 


1 


0 


0 


Explicating goals/procedures 


0 


1 


0 


0 


Applying measurement tools 


1 


0 


0 


3 


Locating/maintaining info 


0 


1 


0 


0 


Using equipment/systems 


0 


0 


0 


0 


Running task oriented meetings 


0 


0 


0 


0 


Getting others to perform 


0 


0 


0 


0 


Adaptation to situation/demands 


0 


0 


0 


0 


Taking another’s perspective 


0 


1 


1 


0 


Total Skills per Product Focus 


5 


38 


18 


9 
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TABLE 12 



Relation of Sensitivities to Product Catej^ ries 





Product Focus 


Categories of Sensitivities 


Research 


Development 


Diffusion 


Evaluation 


Values of self and others 


0 


0 


2 


0 


Capabilities and limitations 


1 


9 


0 


0 


Needs of self and others 


0 


1 


2 


0 


Interactions of self and others 


1 


1 


0 


0 


Context of subject matter 


0 


0 


0 


0 


Worth in disciplines/methods 


0 


0 


0 


0 


Context of objectives 


0 


1 


0 


0 


Worth in objectives 


0 


0 


0 


0 


Awareness of alternatives 


0 


1 


0 


0 


Awareness of structure 


0 


0 


1 


0 


Awareness of method 


0 


0 


0 


0 


Role of catalyst /synthesizer 


0 


0 


0 


0 


Language barriers 


0 


0 


0 


0 


Reality in goal setting 


0 


1 


1 


0 


Degrees of freedom to deviate 


0 


0 


1 


0 


Existing value systems 


0 


1 


0 


0 


Personality of others 


0 


1 


0 


0 


Potential conflict of interest 


0 


1 


0 


0 


Supportiveness required 


0 


0 


0 


0 


Unstated obligations 


0 


0 


0 


0 


Limitations of analyses/data 


0 


0 


0 


0 


Responses of target audiences 


0 


0 


0 


2 


Cost/benefit factors 


0 


0 


0 


0 


Sources of error 


0 


1 


0 


0 


Individual differences 


0 


0 


0 


0 


Recognition of data needed 


0 


0 


0 


0 


Acceptability of output appearance 


0 


1 


0 


u 


Admitting error/adapting 


0 


1 


0 


0 


Willingness to experiment 


0 


0 


0 


1 


Total Sensitivities per Product Focus 


2 


20 


7 


3 



-49 
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Chapter IV: Details on Each Management Responsibility 



This chapter contains (a) for each interviewed management respon- 
sibility a listing of all the standards, tasks, and enablers identified 
during interviews; (b) a summary of citation within QOch category of 
standards and tasks (the enablers are included in summary Tables 5, 6, 
and 7 in Chapter III); and (c) the relationships of standard and task 
categories to product categories by frequency. 

The following are listings of production management responsibilities 
and environmental management responsibilities. 



Production Management Responsibilities 

The management responsibilities detailed in this section are those 
responsibilities that relate directly to management of production 
activities for the project . 

Listing of standards , tasks , and enablers . Each management heading 
is followed by a brief annotation containing: (a) an indication of the 

duration of the responsibility within the duration of the project; 

(_b) whether the responsibility was being performed at time of observation; 

(c) a description of the responsibility; (d) the level (foundational, terminal, 
intermediate, or focal) of the responsibility; and (e) the significant 
relationship(s) with other responsibilities in the project. 

Following the annotation is a brief statement of the interviewee’s 
work focus to the responsibility, and then the standards, tasks, and 
enablers listed under their separate subheadings to identify: 

1. The standards by which the responsibility is judged or 

controlled. 

2. The tasks involved in fulfilling the responsibility. 

3. The enablers necessary for fulfilling the responsibility 

stated in the form of (a) knowledges, (_b) skills, and 

(O sensitivities to be possessed by project personnel. 



PM-59: Product Specifications 



This responsibility was initiated at the start of the project. It 
is ongoing throughout the duration of the project and was operating at 
the time of observation. It is a judging of a product’s acceptability 
by review of subcontractor products to insure inclusion of all relevant 
information. This is a Level III (intermediate) responsibility. By 
allowing for quality control and by setting production parameters, it 
relates to Quality Control Mechanisms (EM-63) and Manage Production of 
Documents (PM-65) . 




-5i . 
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Work Focus of the Interviewee to this Responsibility: 

This responsibility was interviewed around once. The interviewee 
performed the responsibility with assistance from a guidance group. 



Responsibility Standards: 

1. Concensus within guidance group that document contains all 
relevant items. 

2. l^en all the information that has come to me (Project Director) 
is included, (see Task 1, below) 



Responsibility Tasks: 

1. Insure that instructional materials information desired by 
librarians and others within prime contracting agency is 
made available to subcontractor. 

2. Determine means of handling information about materials. 



Enablers of Responsibility: 

1. Skill in writing behavioral objectives and being able to 
recognize the quality of objectives statements. 

2. Skill in making decisions quickly. 



PM-65: Manage Production of Documents 



This responsibility is ongoing throughout the duration of the project 
and was operating at the time of observation, involving decisions on format 
inclusions/exclusions and the appropriateness of document contents. This 
is a Level III (intermediate) responsibility. The details specified below 
are carried out by the subcontractor staff and directly relate to Quality 
Control Mechanism (EM-63) and Product Specifications (PM-59). Both the 
latter responsibilities are those of the prime contractor. 



Work Focus of the Interviewees to this Responsibility; 

This responsibility was interviewed around twice. The interviewees 
performed the responsibility within the subcontractor staff. 
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Responsibility Standards: 

1. Function that had to be performed was described. 

2. The data elements were necessary to support the function 
that was performed. 

3. The relationships make logical sense because nothing was 
excluded from documents. 

4 . Were some things excluded? 

5. The document indicated from where the required information 
was going to come. 

6. Output of system was compatible with other existing systems. 

7. Intended usage was explicity stated. 

8. The system approaches are comparable to existing systems 
in philosophy and concepts. 

9. The system can be maintained by existing staff. 

10. The system can be understood by educators. 

11. There is full coverage in document of costs for collecting 
and maintaining data. 



Responsibility Tasks: 



1. Decide spacing of the document, e.g., single or double 
space. 

2. Decide printing specifications, e.g., single or double 
side. 

3. Determine layout and format of a document. 

4 . Select proper and appropriate illustrations for a 
document. 

5. Insure an adequate table of contents for a document. 

6. Determine if document contains all major areas such as 

inputs , outputs, etc. 

7. Determine if document includes all of the major inputs. 

8. Determine if the inputs are verified by data. 

9. Determine if the inputs were secured from field analysis. 

10. Judge if results of documents are leamable by users. 

11. Determine if the document includes a functional descrip- 
tion . 

12. Monitor adherence of requirements specifications to 
proposal specifications . 

13. Determine if output section of the document is complett . 

14 . Assign responsibility for production to staff members . 

15. Determine if all sections of the documents and the require- 
ments are related. 



Enablers of Responsibility: 

1. General knowledge of educational establishment philosophy, 
practice, and procedures . 

2. Knowledge of how the computer programs operate. 

3. Knowledge of different computer programming languages, 
including their advantages and disadvantages. 

4. Knowledge of expertise within staff. 
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5. 

6 . 

7. 

8 . 
9. 

10 . 

11 . 

12 . 

13. 



Knowledge of Program Evaluation and Review Technique 
(PERT) procedures and interpretation. 

Skill in being precise rather than general. 

Skill in dealing with particulars, details, and minutiae. 
Skill in data processing techniques. 

Skill in analysis of problems. 

Skill in being able to specify the problem. 

Skill in scheduling and following up. 

Sensitivity to needs of final user or system. 

Awareness (inhouse) of how close one can cut the dead- 
lines and push the system. 



Environmental Management Responsibilities 

The management responsibilities detailed in this section are those 
responsibilities that relate to management and maintenance of the project 
environment . 

Listing of standards, tasks, and enablers . The presentation of the 
materials in this section is in the same format as the preceeding. 



EM-57 : Staff — hiring 



This responsibility was accomplished at the initiation of the project 
and was not in operation at the time of observation. The responsibility 
heading is self-descriptive. This is a Level III (intermediate) respon- 
sibility. There are no significant or unique relationships to report 
between this responsibility and others. 



Work Focus of the Interviewees to this Responsibility: 

This responsibility was interviewed around twice. Both interviewees, 
one in the prime contractor staff and the other in the subcontractor staff, 
are directly responsible for hiring. 



Responsibility Standards: 

1. Feedback indicator: The creative response of staff members 

to problems. 

2. The national reputation of individuals. 

3. The opinion of others. 

4. The logic of the argument that staff is adequate. 

5. The good reputation of the agency with the sponsor. 
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6. The concensus of the guidance committee. 

7. Feeling that staff understands project expectations. 

8. Staff members are compatible with other members of the project 
staff . 

9. Staff member accomplished and/or completed tasks or products. 

10. The product (software and/or hardware) that an individual produces 
has accomplished its purpose or works. 



Responsibility Tasks: 



1. Inquire through colleagues and friends to get leads or sugges- 
tions of candidates . 

2. Create descriptive phrases as to the kind of candidate you are 
looking for to hire. 

3. Review resumes of identified candidates. 

4. Inquire as to potential staff member’s availability — ask if 
he/she is available for hiring. 

5. Approach candidate to find out if he is interested in the job. 

6. Make an offer to the candidate. 

7. Justify the need for staff — type and number — to the department 

manager . 

8. Notify department manager to indicate that a potential staff 
member has been identified. 

9. Secure department manager’s approval to employ an individual. 

10. Inquire through manpower data file (computerized vitae file 
of company employees; to identify potential candidates. 

11. Determine technical need required to perform system analysis. 

12. Determine technical competence of existing staff to perform 

system analysis. 

13. Determine if qualified people on staff are available for project 
work. 

14. Determine if qualified personnel can do systems analysis with 
emotional detachment. 

15. Seek a person who enjoys working in this type of project. 

16. Decide to subcontract major part of project because of lack 
of inhouse experts. 

17. Publicize need for subcontractor to perform project. 

18. Select subcontractor on basis of price and reputation. 

19. Select personnel to judge subcontract bid on basis of feasibility. 

20. Negotiate final price of subcontract: with sponsor and subcont ac- 
tor. 



Enablers of Responsibility: 



1. Personal knowledge of potential staff — capabilities. 

2. Know how the agency writes up a personnel evaluation in terms 
of style. 

3. Knowledge of competencies of present staff. 

4. Knowledge of workload of present staff# 

5. Knowledge of possible replacement of staff. 

6. General knowledge of computers. 
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7. Knowledge of national reputalions of people. 

8. Knowledge of where to search for experts. 

9. Knowledge of people holding positions parallel to present 
openings . 

10. In reading (what to look for) a personnel resume. 

11. Sensitivity to criticalness of staff’s present job positions. 

12. Sensitivity to desire and interest of individual in the 
project. 

13. Sensitivity to capacity of those who promise too much. 



EM-58: Budget 



This responsibility is ongoing throughout the duration of the 
project and was in operation at the time of observation. The responsi- 
bility heading is self-descriptive. This is a Level III (intermediate) 
responsibility. There are no significant or unique relationships to 
report between the responsibility and others. 

Work Focus of the Interviewees to this Responsibility: 

This responsibility was interviewed around twice. Both inter- 
viewees, one in the prime contractor staff and the other in the sub- 
contractor staff, are directly responsible for budget management. 



Responsibility Standards: 

1. Concensus by budget experts. 

2. Items are in line with other budgets. 

3. Met the predicted budget or undercut the cost estimates. 

4. Customer says they are happy with the outcome. 

5. Customer funds you with a follow-on project. 




Responsibility Tasks: 

1. Determine travel needs of project staff. 

2. Other recommendations about cost estimates from budget experts 
in department. 

3. Discuss budget figures with sponsoring organizations. 

4. Obtain recommendations about cost estimates from technical 
support group. 

5. Determine realistic costs within the proposal — especially on 
a fixed-price contract. 

6. Determine categorical cost estimates within the budget. 

7. Update budget categorical cost estimates — periodically through- 
out duration of the project. 
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8. Prepare a detailed spending plan — amplification of the 
proposal budget. 

9. Write a monthly budget report — usually a gross or general 
report of what was spent and for what purposes. 

10. Approve spending of funds, e.g., travel cost/time logs. 

11. Review monthly financial statements from the agency business 
office to determine amount of variance in relation to the 
predetermined spending plan. 

12. From the review of the financial statement, isolate or pinpoint 
trouble areas in spending — overspent and/or overestimated areas. 

13. Reconcile problem of inaccurate expenditures encumbered. 



Enablers of Responsibility: 

1. Know rudimentary mathematics. 

2. Know content area within proposal to plan costs. 

3. Know how to read company financial statements — the forms. 

4. Know how to realistically estimate resources relative to work 
that has to be done on the project. 

5. Knowledge of available budget experts. 

6. Sensitivity to mistakes made in past. 

7. Sensitivity to one's own ability and self-confidence. 

8. Confidence in the judgment of experts. 



EM-60: Staff Welfare 



This responsibility is ongoing throughout the duration of the 
project and was in operation at the time of observation. The responsi- 
bility heading is self-descriptive. This is a Level III (intermediate) 
responsibility. There are no significant or unique relationships to 
report between this responsibility and others. 



Work Focus of the Interviewees to this Responsibility: 

This responsibility was interviewed around once with the interviewee 
having the responsibility. 



Responsibility Standards: 

1. Information from secretary indicates personnel satisfaction. 

2. Happy faces around the office. 

3. Personnel are silent and moody. 
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Responsibility Tasks; 

1. Write clear job descriptions for each staff position. 

2. Develop an organization in which people can openly work with 
each other. 

3. Assign clear lines of authority. 

A. Monitor staff awareness of their individual responsibilities. 

5. Listen to subordinates about personal as well as project 
problems. 

6. Allow staff the freedom to create as well as to solve problems. 

7. Insure job (not financial) security of staff by "permanent” 
assignments. 

8. Listen to the complaints of the staff. 

9. Inform staff of all project activities. 

Enablers of Responsibility: 

1. Skill in listening to others. 

2. Sensitivity to the needs of other people. 

3. Sensitivity to the problems people have. 

4. Sensitivity in keeping calm during crisis. 



EM-61: Adequate Intra ^Interagency Relationships 





This responsibility is ongoing throughout the duration of the project 
and was in operation at the time of observation. Due to the organiz- 
ational profile of this project with a prime contractor being dependent 
upon the subcontractor as a production agent, this responsibility is 
highly important to the project. This is a Level III (intermediate) 
responsibility. This responsibility is linkeu to the Project Management 
Decision Structures (EM-62) through the Detailed Project Plan by the 
nature of the project organization. Several committees review the 
products of the project, whic’^ is closely monitored by the prime contractor 
staff, and route comments into the subcontractor production staff. Also, 
the great numbers of potential users of this project's products place 
much importance on this responsibility. 



Work Focus of the Interviewees to this Responsibility: 

This responsibility was interviewed around twice. Both of the 
interviewees, one in the prime contractor staff and the other in the 
subcontractor staff, perform the responsibility. 
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Responsibility Standards : 

1. People are cooperative without pressure. 

2. There is feedback from the user agency, 

3. User agency is persuaded of your point of view. 



Responsibility Tasks: 

1. Establish rapport with each individual that has more than 
transitory contact with the project to maintain awareness of 
concerns . 

2. Talk to project (user agency) people to get a feel for how 
much they know and how it relates to the project. 

3. Deliver and explain all documents to the customer (user agency). 

4. Inform the staff about all activity going on within project. 

5. Accommodate user agency by giving presentations relative to 
the project to various groups. 

6. Provide user ^y^^eiicy some extra products (promotional). 

7. Report status of project to national advisory board on regular 
basis . 

8. Inform state coranittee about status of project at regular 
intervals. 

9. Inform the sponsoring agency about the status of the project 
at regular intervals. 

10. Participate in interagency meetings for informing all parties 
concerned. 

11. Explain to the customer what is expected in terms of feedback. 

12. Decide who should be given what information about the project. 

Enablers of Responsibility: 

1. Know what the customer’s work is about and how it relates to 
the project. 

2. Knowledge of who makes which decisions. 

3. Knowledge of who needs what information. 

4. Ability to inspire the customer’s confidence in Project Director 
and staff. 

5. Ability to understand the customer’s jargon. 

6. Ability to speak in front of a group of people. 

7 . Ability to answer questions under pressure . 

8. Insight into an individual’s personality — can figure out what 
pleases him. 

9. Sensitivity about human relationships. 

10. Keeping aware of all activities going on. 
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EM-62: Project Management Decision Structure 



This responsibility was initially established very early in the 
project and is ongoing throughout its duration. It was in operation 
at rhe time of observation. The responsibility heading is self-descrip- 
tiv . This is a Level III (intermediate) responsibility. This respon- 
sit lity is rooted in the philosophy of the project and feeds into all 
thw others. 



Work Focus of the Interviewees to this Responsibility: 

This responsibility was interviewed around once with the interviewee 
having the responsibility. 

Responsibility Standards: 

1. I feel confident in decision mad^i. 

2. People are happy with decisions made. 



Responsibility Tasks: 

1. Decide who should make which decisions. 

2. Decide who should make decisions about priorities. 

3. Review national board opinion, but do not allow them decision- 
making power. 



Enablers of Responsibility: 



1. Knowledge of systems concepts. 

2. Skill in making decisions. 

3. Sensitivity to one's own limitations. 

4. Confidence in other people's judgment. 



EM*'63: Quality Control Mechanism - Schedules 



This responsibility is ongoing throughout the duration of the 
project and was in operation at the time of observation. The respon- 
sibility heading is self-descriptive. This is a Level III (intermediate) 
responsibility. The Detailed Project Plan, developed at the initiation 
of the project, feeds into this responsibility. 
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Work Focus of the Interviewee to the Responsibility: 

This responsibility was interviewed around once. The interviewee 
(member of prime contractor staff) performed the responsibility as a 
monitor of production by the subcontractor. 



Responsibility Standards : 

1. Concensus of guidance committee. 

2. Work accomplished according to plan. 

3. Has no apparent discrepancies. 



Responsibility Tasks: 

1. Monitor expenditure of funds by statf. 

2. Monitor expenditure of time by staff on various parts of the 
project . 

3. Monitor adherence to project policy by all members of the 
staff . 

4. Recommend corrective action tor deviations from schedules as 
necessary . 

5. Estimate if proposed timeline is realistic for job to be done. 

6. Counsel with various members of the staff a.bout their problems. 

7. Monitor the flow of information between staff and subcontrac- 
tor staff. 

8. Plan and conduct staff meetings. 

9. Inform all concerned parties about decisions made within 
project . 



Enablers of Responsibility: 

1. Knowledge of what PERT is and how it operates. 

2. Knowledge of various management techniques. 

3. Knowledge of educational establishment, philosophy, people, 
and procedures. 

4. Knowledge of field setting staff organization. 

5. Skill in developing flow charts. 

6. Sensitivity to field setting protocol and prr^cedures . 

7. General knowledge of data processing procedures. 



PM~67; Staff - Work Scheduling 



This responsibility is ongoing throughout the duration of the 
project and was in operation at the time of observation. The responsi- 
bility heading is self-descriptive. This is a Level III (intermediate) 
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responsibility. The Detailed Project Plan, developed at the start of 
this proj ect , feeds into this responsibility . 

Work Focus of the Interviewee to this Responsibility: 

This responsibility was interviewed around once. The interviewee 
(member of subcontractor staff) performed the responsibility to insure 
the production process. 

Responsibility Standards: 

1. Staff has produced products within time constraints. 

2. Staff has produced product within budgeting constraints. 

3. Individual has produced as much or more than was expected. 

4. Individual has carried out tasks in an exemplary manner. 

Responsibility Tasks: 

1. Assign tasks or pieces of work to staff members. 

2. Establish schedule time lines for task completion. 

3. Identify other personnel (within the agency) to carry out 
tasks — agency specialists that might be needed periodically. 

4. Arrange to use (time of) personnel identified in the agency. 

5. Review the work or staff performance of tasks. 

6. Consider work loads in the assignment of tasks. 

Enablers of Responsibility: 

1. Have technical competence (relative to technical knowledge 
within content area of the project) so staff is confident in 
you. 

2. Knowledge of personnel psychology. 

3. Knowledge of interview techniques. 

4. Knowledge of content and specialty (related to project content) 

areas . 

5. Ability to communicate well with staff. 

6. Skill of writing a task assignment to insure understanding 
by staff of the task. 

7. Ability to reduce technical jargon to level audience under- 
stands . 

8. Have the maturity to admit your mistakes. 

9. Awareness of each staff member*s technical competence. 

10. Sensitivity to each staff member *s personal problems. 

11. Sensitivity to each staff member’s capacities in terms of 

workload and pressures they can handle. 
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Summaries of Management Responsibilities Data 

Tables 13 and 14 are summaries by frequency of citation within cate- 
gories of management responsibility standards and tasks. 

In Table 13, the categories of standards with the highest frequencies 
are ’‘personnel satisfaction**, “operational demands equate with estimate/ 
projections,” and ’’cost-benefit relationships are acceptable.*' In this 
project, management standards are based on the total staff judgment or 
the judgments of committees that production rates are within knowledgeable 
estimates and that products are efficient. 

In Table 14, the categories of tasks with the highest frequencies 
reflect the emphasis within this project on production control and 
environmental management in terms of staffing, securing consultants, 
and dissemination cf product and project information. 
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TABLE 13 

Frequencies of Citation of Standards for Management 
Responsibilities in each Standards for Management 
Responsibility Category 



Categories of Standards for Management Responsibilities 

Personnel cooperate 
Personnel are satisfied 
Minimum correction required 
Deadlines are met 
Acceptable level of output 

Work structure is efficient 
An expected activity occurs 
Staff contributions accepted 
Outside contributions accepted 
Maximum possible participation 

Cost consistent with estimates 

No obvious omissions 

Work conducted within budget 

No felt deficiencies 

Tasks perceived and acted upon 

External cooperation gained 
External enthusiasm evident 
Desired personnel obtained 
Adequate reputation with sponsor 
Performance respected 

Follow-on proposals are funded 
Feedback occurs 
Project view accepted 
Cost acceptable for benefits 

Total Number of Standards Cited for Management 
Responsibilities 




.6 4 f 



Frequency 
of Citation 

1 

4 

0 

2 

2 

0 

0 

0 

0 

0 

3 

0 

2 

0 

1 

1 

0 

1 

1 

1 

1 

1 

1 

3 
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TABLE 14 

Frequencies of Citation of Tasks 
±1 Each Tasks Category 



Frequency 

Categories of Management Responsibilities Tasks of Citation 

Procuring professional staff 17 

Effecting accountability 34 

Procuring systems/services 9 

Effecting quality control 13 

Maintaining job satisfaction 6 

Facilitating growth of staff 0 

Enhancing physical environment 0 

Maintaining equity among staff 1 

Facilitating relationships 4 

Effecting information flow patterns 0 

Diffusing information within project 8 

Diffusing information beyond project 5 

Effecting decision mechanisms 1 






Total Number of Tasks Cited 
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Interaction of Management Responsibility Data 

Tabulations of management responsibility standards and tasks by 
product focus are found In Tables 15 and 16. The category of responsi- 
bility focus In both tables containing the highest frequencies Is that 
of development. In addition, some effort In diffusion can be noted 
In Table 16. 
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TABLE 15 



Relation of Standards for Management 
Responsibility to Product Categories 



Categories of Standards for 
Management Responsibilities 


Responsibility Focus 


Research 


Development 


Diffusion 


Evaluation 


Personnel cooperate 


0 


1 


0 


0 


Personnel satisfied 


0 


4 


0 


0 


Minimum correcting required 


0 


0 


0 


0 


Deadlines are met 


0 


2 


0 


0 


Acceptable level of output 


0 


2 


0 


0 


Work structure is efficient 


0 


0 


0 


0 


Expected activity occurs 


0 


0 


0 


0 


Staff contributions accepted 


0 


0 


0 


0 


Outside contributions accepted 


0 


0 


0 


0 


Maximum possible participation 


0 


0 


0 


0 


Costs consistent with estimates 


0 


3 


0 


0 


Obvious omissions 


0 


0 


0 


0 


Work conducted within budget 


0 


2 


0 


0 


No felt deficiencies 


0 


0 


0 


0 


Tasks perceived and acted upon 


0 


1 


0 


0 


External cooperation gained 


0 


1 


0 


0 


External enthusiasm evident 


0 


0 


0 


0 


Desired personnel obtained 


0 


1 


0 


0 


Adequate reputation with sponsor 


0 


1 


0 


0 


Performance respected 


0 


1 


0 


0 


Follow-on proposals are funded 


0 


1 


0 


0 


Feedback occurs 


0 


1 


1 


0 


Project view accepted 


0 


1 


1 


0 


Costs acceptable for benefits 


0 


1 


0 


2 


Total Number of Standards Cited 
for Management Responsibilities 


0 


23 


2 


2 





^^0 



yuc5 



TABLE 16 

Relation of Management Responsibility 
Tasks to Product Categories 



Categories of Management Tasks 


Responsibility Focus 


Research 


Development 


Diffusion 


Evaluation 


Procuring professional staff 


0 


17 


0 


0 


Effecting accountability 


0 


29 


5 


0 


Procuring systems/ services 


0 


9 


0 


0 


Effecting quality control 


0 


9 


4 


0 


Maintaining job satisfaction 


0 


6 


0 


0 


Facilitating growth of staff 


0 


0 


0 


0 


Enhancing physical environment 


0 


0 


0 


0 


Maintaining equity among staff 


0 


1 


0 


0 


Facilitating relationships 


0 


2 


0 


0 


Effecting information flow patterns 


0 


3 


1 


0 


Diffusing information within project 


1 


6 


0 


1 


Diffusing information beyond project 


0 


5 


0 


0 


Effecting decision mechanisms 


0 


1 


0 


0 


Total Tasks per Responsibility Focus 


1 


88 


10 


1 
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Chapter V: Supplementary Data 



Product Frequency of Occurrence by Category 

Table 17 presents a summarization of products and management respon- 
sibilities by the research, development , diffusion, and evaluation focus . 
These results, as are all the summary results, are based upon the coding 
categories under development in the present study • 

The reader will note that under product focus the greater frequencies 
occur in ^development followed by **dif fusion. ** In "development” the 
greater frequencies occur in all three categories of output orientation. 
But in *Miffusion” the highest frequencies occur in the production 
orientation. This reflects the subcontractor's efforts in dissemination 
of its products to the prime contractor, as well as the preparation of a 
journal article and workshops. The highest frequencies occur in output 
orientation under production and environmental management. The former 
reflects the effort of the prime contractor. The profile that emerges 
is one of primarily a development effort with a production orientation. 
Emerging as supportive to these is the diffusion effort and the environ- 
mental management orientation. 

Note the relatively high frequency occurring in the evaluation focus 
under production orientation. Most of the initial production effort in 
this project was the evaluation of existing automated information systems 
in use. 



TABLE 17 

Frequencies of T 3 rpes of "Responsibilities” 
(Products and Management Outcomes) 



O 

ERIC 





Output Orientation 


Product /Responsibility 
Focus 


Production 


Production 

Management 


Environmental 

Management 


Product 
Focus 
' jtals 


Research 


1 


0 


0 


1 


Development 


18 


7 


17 


42 


Diffusion 


11 


3 


1 


15 


Evaluation 


10 


1 


0 


11 


Output Orientation 
Totals 


40 


11 

.1 


18 

i 


69 
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Summary of Staff Backgrounds 

The information in this section is based on only the responses of 
the eight staff members of the AIMS project who were formally interviewed. 

Months on the project . At the time of observation the project was 
entering the seventh month of its work, having begun June 1, 1970. The 
duration of time on the project in months per each interviewee has the 
following distribution. The duration of the project is 12 months. 



Number of staff 



Months of Project Experience 
0-1 2-6 7 - 12 

0 2 6 



Current percentage of time assigned to project . At the time of 
observation the eight staff interviewed had the following percentage 
of full-time equivalency (FTE) assigned to this project. 



Number of staff 



Percentage of FTE 



25% or less 30% to 60% 65% or more 



The two staff members with assignments of 65% or more filled 
assistant management positions in the project. One in the prime contrac- 
tor’s office and the other in the subcontractor’s office. 

Highest degrees attained . Table 18 identifies the number of degrees 
held by the staff at each level and the areas of specialty of each degree. 

National professional memberships . The following professional organ- 
izations are represented by one or more of the interviewed staff having 
membership(s) in one or more of the organizations. 



1 . 



2 . 



3. 



American Society of Information Science 
Society of Technical Writers and Publishers 
National Microfilm Association 
American Library Association 
Special Library Association 
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TABLE 18 

Frequency of Degrees per Specialty Area 



Highest Degree 



Specialty Area 


Bachelor’s 


Master’s 


Specialist 


Education/ teaching 
Business Administration 


i 

1 

1 

1 


1 




Information Science 
(Do cumentation) 




; 1 

1 




Library Science 




1 

2 


1 


Language 


1 






Advertising & Marketing 


1 






Totals 

1 


3 

— ‘ 


4 




1 



Prior work experience * Table 19 displays the distribution of years 
of work experience of the eight staff members interviewed within eight 
setting categories. 
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TABLE 19 



Distribution of Staff Work Experience per Work Setting Category 



Amount of Experience 



Work Setting Category 


No 

Experience 


Less 
than 
1 year 


1-4 

years 


1 

5 or more 
years 


In R,D,D. or E Work 


1 


0 


2 


5 


In Administrative Work 


0 


1 


1 


6 


In College Teaching or Research 


7 


0 


1 


0 


In Public Schools 


4 


0 


1 


3 


In State or National Education 
Agencies 


7 


0 


1 


0 


In R & D Centers 


5 


0 


1 


2 


In Present Organization (may be 

concurrent with other areas above) 


0 


0 


3 


5 


In Other Work Settings 


4 


0 


0 


4 



Summary of Interviewee Responses 

Present position requirements * Four questions asked of the eight 
interviewees are stated below with their responses. The responses are 
listed by position. 



Question 1: ”What specific knowledges and skills does (your) positior, 

require?** 



Principal Investigator and Project Director 



O 
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1. Administration skills. 

2. Project development and management skills. 

3. Knowledge of state and national trends in field studies. 

4. Knowledge of people with expertise in field studied. 
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Deputy Director and Project Monitor 

5 • Management techniques that include organizational patterns . 
6# Knowledge of accounting and budgeting. 

7. Contract procurement procedures. 

8. Systems analysis (PERT chart and flow-charting). 

9. Data processing. 



Planning and Guidance Panel Member 

10. Problem analysis and systems design. 

11 . Computing system hardware , software , and languages. 

(This position serves as a computer f ' stems consultant 
position to the Project Director.) 



Project Manager (subcontractor) 

12. Knowledge of computers and data processing. 

13. Knowledge of librarianship . 

14. Writing skill. 

15 . Personnel management ski 11 . 



Assistant Project Manager (subcontractor) 

16. Knowledge of system design concepts. 

17. Writing skills. 

18. Administrative skills. 



Senior Systems Analyst (subcontractor) 

19. Systems analysis ^nd design. 

20. Publications. 

21. CommunicaticaS) system analysis and design skills. 



System Testing and Library System Specialist (subcontractor) 

22 . Requirements analysis . 

23. System design. 

24. Programming « 

25. Information management. 



Question 2: "How many years of work experience does (your) position 

require in educational research, development, diffusion, 
and/or evaluation?" 
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]. Five inlerviewees indicated that one to four years of 

experience was required. Their positions were Principal 
Investigator and Project Director, Deputy Director, 
Assistant Project Manager (subcontractor) , Senior System 
Analyst (subcontractor), and System Testing and Library 
Systems Specialist (subcontractor) . 

2. Three interviewees indicated that five years or more 

experience was required. These positions were Planning 
and Guidance Panel members (computer expert), Project 
Manager (subcontractor), and Senior Systems Analyst (sub- 
contractor) . 



Note in Table 19 that one interviewee had no experience in R,D,D,&E 
work', while two had beivreen one and four years experience and five had 
at least five years experience. 



Question 3: '*How many years work experience does (your) position 

require in administration or management?’’ 



1. One interviewee indicated no administrative experience 
necessary. (This position was production orientated. ) 

The position was System Testing and Library Systems Spec- 
ialist (subcontractor) . 

2. One interviewee indicated that some, but less than a year’s 
experience in administration was required for this position. 
The position title was Assistant Project Manager (sub- 
contractor) . 

3. Five interviewees indicated that one to four years exper- 
ience was important to their positions. The position titles 
are Principal Investigator and Project Director, Deputy 
Project Director, Project Manager (subcontractor), and 
Senior Systems Analyst (subcontractor) . 

4. One interviewee indicated that five years or more exper- 
ience in administration was important to his position. 

The position title was Planning and Guidance Panel Member 
(computer expert). 



Question 4: ’’Academically (your) position requires which degree?” 
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Degree Degree 

Interviewee Interviewee Indicated 



Position title 


Presently holds 


Position Required 


Principal Invest igator and 
Project Director 


master ’ s 


master ’ s 


Deputy Director 


master ’ s 


master’ s 


Planning and Guidanc^i 
Panel Member 


bachelor ’ s 


bachelor *s 


Project Manager 


specialist 


specialist 


Assistant Project Mar ager 


bachelor *s 


bachelor ’ s 


Senior Systems Analyst 


bachelor ’ s 


bachelor ’ s 


System Testing and Library 
Systems Specialists 


master* s 


none 


Senior Systems Analyst 


master’ s 


doctorate 



Support resources . The support resources, both service and equip- 
ment, used by the p^^rsonnel on this project were: 

1. Support services used: 

Printing. 

Duplication services. 

Photography. 

Art work and illustrations. 

Secretarial service . 

Typing. 

Library seirvices. 

Computer analysis services. 

Computer program wi'iting. 

Audio-visual aids and devices . 

Customer supervisory personnel. 

2. Support equipment used: 

Dictating or recording equipment. 

Desk calculator. 

Remote computer terminal . 

Onsite computer. 

Keypunch machine. 

Photographic equipment. 

Typewriter. 
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Classifications of Outputs 

As the Oregon Studies evolved it became evident that outputs could 
be categorized in terras of a number of variables. Among them are (^) 
Structure (product, event, or condition), (b) Function (policy setting, 
management, or production), (O Level (focal, component, or facilitating), 
(^) Character (knowledge, technology, implementation, or information), 
and (jb) Stage of Completion. These five schemi are represented in 
Table 20 for each project output identified, with frequencies summarized 
for each category. Table 20 has been added to this profile subsequent 
to the profile's original writing. 
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Class 1 1 Icat ions of Output Ch«ract®rlnt lot 







Ov 




t r r 1 1 1 1 . ^ 




Project Outputs 








li.tr.Sk t er 






_S_t_nic^u t _• 


Function 


1 rvf 1 


onlvji 


Completion Stage 


No. label 


P < c 


rn re p 


^ - 




V 2 i 4 5 b 


P-46 System Design Alternatives Evaluation 


X 


X 


X 


X 


X 


Procedure 












P-47 System Design Concepts 


X 


X 


X 


X 


X 


•P-48 Pilot System of Autotnated Instruct lonal 


I 


X 


X 


X 


X 


Materials Handling System 












•P -49 Preliminary System Description 


I 


X 


X 


X 


X 


P-SO System Description 


I 


X 


X 


X 


X 


PM'51 Checklist for Design Review and Con“ 


X 


X 


X 


X 


X 


currcnce Package 












•P-52 Journal Article 


I 


X 


X 


X 


X 


P.M-53 loformatlon Woikshop 


I 


X 


X 




X 


•EM-5A Sub-contractor to Prime Contractor 


X 


X 


X 


X 


X 


Monthly Report 












P-55 Two other Cities Site Visit Trip Reports 


X 


X 


X 


X 


X 


P-56 LAUSD Site Visit Reports 


X 


X 


X , 


X 


X 


*EM-57 Stsff - Hiring 


X 


X 


X 




X 


•EM- 58 Budget 


X 


X 


X 


X 


X 


•P.M-59 Product Specifications 


X 


X 


X 


X 


X 


•EM-60 Staff - Welfare 


X 


X 


X 




X 


•EM-61 Adequate Inter/lntra Agency Relationships 


X 


X 


X 




X 


*EM-bii Project Management Decision Structure 


X 


X 


X 




X 


•PM-63 Quality Control Mechanism - Schedules 


X 


X 


X 


X 


X 


•PM-64 Progress Reports 


X 


X 


X 


X 


X 


•PM-65 Managed Production of Documents 


X 


X 


X 




X 


•PM-67 Staff - Work Scheduling 


X 


X 


X 




X 


P-68 Interview Guide (Other Cities) 


X 


X 


X 


X 


X 


EH-69 Halntalned Philosophy of Approach to 


X 


X 


X 




X 


Problem 












Dl-70 Sub-contractor’s Internal Monthly Report 


X 


X 


X 


X 


X 


to Division Head 












•P-71 Audio-vlsual Instructional Materials 


X 


X 


X 


X 


X 


System Requirements 












P-72 Proposal - SDC 


X 


X 


X 


X 


X 


P-73 User Meeds Analysis Docusientat ion 


X 


X 


X 


X 


X 


ClsBsification Frequencies^ 


61 3 5 


1 51 17 


3 13 53 


0 41 4 16 


2 29 9 10 9 10 



* Ths specific output 


charsctsrlBticB are 


identified ss follows: 




Scruc ture 


Function 


Level 


Character 



CoaplatloD Sta^g 



p - produce 
t - event 
c - condltloD 



pa *' policy settiog 
D - foanagement 
p - production 



- focal 

c - component 
f 2 ** facilitating 



k O' knowledge 
t - technology 
1^ Inplementatlon 
I 2 • Infonaatlon 



1 cooplcted over one year ago 

2 - completed 3 to 12 contha ago 

3 - completed within last 3 moa. 

4 - currently in progress 

5 - not yet underway 

6 • on going (continuous) 



Data totals in this table may vary slightly from data in tables reported elsewhere. This la a function of decision rules 
governing classification of outputs having been revised and applied to these data subsequent to the preparation of the 
profils. 
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Chapter VI: Project Dynamics 



Origin o£ the AIMS Project 

The roots of this project can be traced to a State (California) 
Department of Education meeting called to bring together the two fields 
of library (book) and audio-visual (nonbook) instructional materials 
to provide a closer working association between them. Based on discussions 
at that time the development of this project was initiated within the 
Los Angeles Unified School District. 

Early in the planning, the scope of the project was expanded to 
include the areas of textbook and science-center materials, along with 
library and audio-visual. 

It took three years to write and fund this project. Once goals and 
parameters for the project were established, the school district issued 
a request for proposals to several potential subcontractors. The PGP 
in the school district and the National Advisory Committee monitoring 
bodies were established to oversee the production effort of the subcontrac- 
tox*. This was done to assure relevance in the system to be designed to 
the school district needs and to the needs of school districts nationally. 

Within the school district the project was originally in the Division 
of Instructional Planning and Services, but in District reorganization 
it was moved to the Division of Special Services. The District recently 
reorganized to decentralize operations, so that sections are now semi- 
autonomous within Divisions. Consequently there seems to be minimal 
agency-level involvement with this project. The precise nature of 
District involvement was not determined in the study interviews. 



Physical Setting 

The offices of the prime contractor (LAUSD) staff are located in 
downtown Los Angeles. Members of the interview team reported that the 
offices were quite suitable and comfortable. Adjacent to the Project 
Director’s office was a small conference room containing one table and 
bookshelves. Nearby is the library set up for use by teachers and D:*«^l irict 
staff. On the floor below the offices, were the District’s television 
studios. District and Division offices are located elsewhere. 

The offices of the subcontractor (SDC) staff are located in Santa 
Monica. The relative nearness of the subcontractor staff allows for 
frequent meetings with the Project Director and Deputy Director at their 
offices . 

The subcontractor staff is in one of two large, multi-level office 
buildings that house the activities of several hundred people. Due to 
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lUit classified naturi’ of some of the v.»(;rl: ('nnilu('l(‘<l hy Sl)(l, a visitor 
must h(‘ escorted from and to the ri'cent i on area. Also, an emnloyoe. 
must sliow a security card clearnnce when ho passes throup.li the reception 
area even tliouj^li tlio r(*ceptioni.s t may haiow him hy sip, lit and name. 

Each staff member had his own office with the f urn i ture usual ly 
consisting of a desk, two or three chairs, one or two filing cabinets 
(with combination locks), one to three bookshelves, and a work table. 

The equipment observed in the offices included typewriters, calculators, 
dictaphones, telephones, and microfiche readers. Stacks of papers 
seemed to occupy much space in each of tiie offices, but the work space 
seemed adequate. 

The noise level in the offices was judged by tlu' interview team 
as very high due to the heavily traveled streets adjacent to the office 
buildings. However, the interviewees did not seem to notice or appear 
bothered by the noise level. 



Management and Come. unicat ion Process 

The management philosophy of the Project Director Is one of delegating 
responsibility and authority, while acting as a catalytic agent. Crit- 
ical decisions are made when necessary and others are encouraged to accept 
responsibilities and make minor decisions . 

Some of the specific functions the Project Director engages in 
include tracking the production of the subcontractor by conferring with 
the Deputy Director to see whether a document is or is not completed, 
conferring with an expert to determine the content quality of a document, 
seeing that information is transferred from a source to a user and insur- 
ing that action through the Deputy Director, and building or maintaining 
project interface with agencies and people relevant to the project. 

The Deputy Director supports the Project Director hy continually 
monitoring the production rate in the subcontractor * s offices, providing 
follow-through on decisions and requests of the Director, setting up 
meetings , and monitoring the budge t . 

There is a strong interagency reliance or dependence between the 
prime and subcontractor. The prime contractor has a major problem — to 
design and implement a system to handle the District’s ins tructional 
materials operation; the subcontractor has tho technical knowledge and 
skill to assist in solving the problem — to design and develop a system 
to handle the District’s instructional materials operation. Tlie sub- 
contractor, in turn, depends very heavily on information (in terms of 
their design and implementation plans for the system) from the prime 
contractor for successful solution of the problem. 
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Upon completion of each piece or connonent of materials of the 
system by the subcontractor, it is delivered and explained in detail 
to specified prime contract personnel to insure complete understanding 
of the material. This is done so that the? priru' contractor will ho 
ab le : 

1. To determine if the details seem to be realistic 
and workable in terms of the District *s needs. 

2. To provide the subcontractor with necessary feed- 
back in terms of the feasibility of the system 
component , 

The LAUSD is the agency who will ultimately use tlie Instructional- 
Materials-ilandling System in support of solution of the problem, and 
tlie subcontractor warts to make sure LAUSD understands and approves 
of the system in terms of its operability for their use. 

The. style of project management, as onerational Ized by the 
Project Manager, was to run the project in a flexible ranner. Planning 
and assigning tasks was usually done in conjunction \;i th the project 
staff . Once the tasks were identi r jued and described , the project staff 
tended to volunteer themselves to carry out a task or set of tasks based 
on the knowledge of their own capabilities. Roughly 90% of the task 
responsibilities are delegated in this manner. The tasks that are not 
delegated on a volunteer basis are usually reevaluated and reworked by 
the Project Manager for going through the same process again, or, perhaps, 
parcelled out by the Project Manager on a trial basis. 

Planning and decision-making relative to the production of project 
outcomes (products) is done in a combination of democratic and consensus 
processes. The project staff is assembled to discuss the issues; every 
one is free to contribute and to make suggestions, A consensus is usually 
reached. However, the Project Manager has primary responsibility for 
the project and has final authority over any decision. 

The Project Manager and staff are very committed to the project. 

The entire staff operates op an interdependent basis and relies heavily on 
interaction (oral and written) with one another. The Project Manager 
monitors progress of the project very closely and is very concerned — is 
are the rest of the project staff — with quality and excellence of project 
outcomes. There is much interaction, review, and critique of all products 
that are generated and the Project Manager reviews all materials that are 
produced in re lation to the project . It is the philosophy of the project 
staff to go beyond just pleasing the customer or user agency. They want to 
provide the customer with the best possible product within the resource 
limitations . 

Also, the Project Manager personally delivers major products and 
documents to the prime contractor to explain the contents, to clarify 
any possible misunderstandings, and to assist in giving them a set of 
guidelines by which they can critically review the document and provide 
the subcontractor with valuable feedback — the* feedback v/hich faeijitates 
the development of a quality and useful product for the customer. 
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Issues 



Reorganization to decentralize within LAUSD has been a problem in 
the work of this project. This was especially so in the early phase of 
the project when the subcontractor was in the field attempting to evaluate 
LAUSD’ s materials handling practice in order to provide a suitable data 
base from which to design an appropriate system. 

Another problem for the project has been its timing. Apparently 
decisions on funding were not made until just before closing of school 
in the early summer of 1970. Consequently interviewing of school 
personnel was done at a very inconvenient time for them. In addition, 
the necessity to get the interviews done in the short time that school 
would still be in session forced the subcontractor to conduct its 
interviewing before interview guides and questionnaires were fully 
developed. When the questionnaires were sent out many people did not 
respond because an interviewer had already talked with them, and they 
assumed it was not necessary to fill out the questionnaires. However, 
most of the school people eventually did fill out a questionnaire or 
supply the information in some form. 

The late decision on funding also placed rather severe time 
constraints on the production of products in this project. The proposed 
duration for the work was 12 months, starting March 15, 1970 and 
ending March 17, 1971. However, work did not actually start until June 1 
due to the late decisions on funding, but the ending date was not slipped 
back to compensate for the late start. 

In staffing, both the prime contractor and the subcontractor have 
had problems. In the school district all current personnel are so 
tied down there is difficulty in having someone released for a temporary 
position in a project. Hiring a new person for a temporary position is 
not possible either. Early in the project, the subcontractor lost a key 
staff member. Because most of the staff had commitments to other 
projects, they were not always immediately available. 
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Chapter VII: Implications for Training 



The purpose of this chapter is to summarize the information gleaned 
from this project relative to the training needs for professionals in 
educational R, D, D, or E. 

Often staff members of a project are assigned to that project only 
a portion of their total work time. Such is the case in this, the AIMS 
Project. The average percentage of work time assignment over the eight 
staff members interviewed was A0%. The range was from 10% to 100%. Only 
one member was assigned 10% and just two were full time (100%). All staff 
with commitments less than full time concurrently carried other duties or 
assignments to other projects. If this is the reality of educational R, 

D, D, & E, then this should be reflected in the training. 

Among the subcontractor’s staff the concept of teamwork was heavily 
emphasized. This approach seems to reflect the business approach to 
educational development . The relevance of this idea to social innovation 
seems very appropriate. The concept of working staff, each with a well- 
defined area of expertise, contributing a whole part to the final goal seems 
much more appropriate than where a single individual is asked to contri- 
bute a portion to each of the parts in the final goal. The implication 
here for training, at least in the academic setting, is that possibly the 
preparation of a dissertation should be a group effort. Each student 
would contribute a part to the dissertation. He would develop considerable 
expertise and skill in using the skill of others — teamwork. The results 
could be professionals well prepared to enter into broad, long range social 
innovation programs. 

Writing is a skill essential to the work in this project. As one 
interviewee stated, ”We are dependent in most of our work upon writing, 
because this is the product we deliver to a customer. If one cannot 
write, technical skills really are not that valuable." 

In response to questionnaire items concerning the preservice training 
most relevant to their present work, the staff who were intervj '^^ed listed 
the following : 

1. Library science course work. 

2. Writing (English courses). 

3. Experience in systems analysis and design. 

4. Experience in computer-based systems. 

5. Experience in technical writing. 

6. Experience in technological research. 

7. Data processing. 

8. Library administration courses. 

Many of the staff indicated that actual work experience (on-the-job 
training) seemed more valuable to them than their formal academic training. 
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Concerning inservice training to prepare them specifically for 
work on this project, the staff listed the following: 

1. Briefing on computer architecture for the ’70’s. 

2. Training in making technical speeches. 

3. Project briefings. 

4. Project management and control. 

Special note was made to training in making technical speeches by 
one interviewee. Much of the dissemination activity in this project is 
carried out verbally, necessitating skill in effectively presenting pro-- 
ject information. 

When asked via a questionnaire to suggest some areas relative to 
their work within this project where formal training could be specialized 
the staff listed the following: 

1 . Pro j ect management . 

2. System design. 

3. Library work. 

4. Computing science. 

5. Mathematics. 

6 . Industrial engineering . 

7. Contract procurement procedure. 

8. Accounting and budgeting. 

9 . Management techniques to include organizational patterns. 

In addition to the questionnaire, each staff member interviewed was 
asked for his training suggestions relevant to his job on the project. 

The following is a listing of those suggestions: 

1. Attending conferences or conventions to develop acquaintance 
with people working in the area of study interested in. 

2. Management techniques such as development and use of flow- 
charts, PERT charts , management principles , and development 
of skill in planning ahead. 

3. Computer programming . 

4. Sensitivity training for managers where they get a chance to 
experience people's reactions to themselves and their ideas. 

5. Personnel work — management skills related to people. 

6. Interview technique — how to carry out an employment interview. 

7. Problems of budgeting a project — how to plan and put together 
a realistic and accurate budget. 

8. Technical writing — how to prepare flow-charts. 
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9. Technical editing — how to Judge whnt level or what language 
is suitable for the reader. 

10. Project 'Simulation exercises to help prepare the trainee for 
such things as identifying potential bottleneck areas in 
conducting a project. 

11. To maintain a “state of the art" awareness of the advances in 
technology — important in system design. 

12. Apprenticeship-- or internship-like arrangement with people and 
organizations involved in educational development to provide 
student with actual work experience. One interviewee suggested 
that this arrangement be in industry rather than education . 

13. Training for students in working under the constraints of time 
and money. Teaching researchers to budget their time and 
spend it as if someone were paying for it. 

14. Informal information gathering — how to conduct an informal 
but focused interview. 

15. Dissertations or theses could be written by committees of 
students. For example, the student would concentrate in one 
or more areas of research, developing a high skill level in 
those areas. For skills or knowledge of other areas he does 
not have, the student would rely on other students or experts 
in those areas. A student’s progress could be judged by how 
many satisfactory hours he had spent in various portions, such 
as project manager, production worker, etc. in one or more 
projects. He would also be judged on his contribution to the 
dissertation or thesis produced. 



A GUIDE TO THE OREGON STUDIES IN EDUCATIONAL RDD&E 



Volume I 

SUMMARY REPORT 

An introduction to and overview of the Oregon Studies as a 
whole. The volume contains an outline of the history of the 
Studies, the rationale around which they were designed, the 
context within which they were carried out, and the 
procedures followec in their execution. It also contains a 
description of the projects selected for study, the rationale 
underlying their selection, the criteria and procedures used 
in their selection, and an overview of the data collected on 
each project. Finally, the volume contains an introduction 
to the “case profiles” that house the data collected on each 
project, the results of all cross-project analyses, and the 
summary recommendations that have been made relative to 
training and the continued study of educational RDD&E 
activities. A brief description of the case study method- 
ology. developed within the Studies, an overview of a 
process whereby investigators may query computer-stored 
data files and original interview statements to obtain 
information bearing upon specific questions relating to 
training, manpower, policy, and work performance, and 
supporting data accompany the volume. 



Volume II 

THE UTERATURS OF EDUCATIONAL RDD&E 

A compendium of existing literature that defines, describes, 
differentiates, or relates the activities labeled educational 
research, development, diffusion, evaluation, and various 
combinations thereof. The articles within the vo^mne are 
introduced as a collection. Linking passages provide an 
interpretive context t oth for individual articles and for the 
sets into which they have been grouped. 



Volume HI 

CONCEPTUAL i^^aMEWORKS FOR VIEWING 
EDUCATIONAL RDD&E 

A collection of papers which provide the conceptual 
underpinnings to the Oregon Stuuies. It contains three 
papers commissioned by the Studies as a basis for con- 
ceptual development, and a paper by staff from Teaching 
Research that describes the conceptual frame that guided 
and grew with the empirical thrust of the Studies. Each of 



the papers is a major document which defines, differ- 
entiates, and relates one or more facets of educational 
RDD&E and provides a supporting rationale for the 
position adopted. Each paper is accompanied by a formal 
critique, and the set of papers is accompanied by an 
introductory and summary critique. 



Volume IV 

PROFILES OF EXEMPLARY PROJECTS 
IN EDUCATIONAL RDD&E 

A collection of twenty case profiles that form the data 
base in the Oregon Studies. Printed in three parts, the 
profiles describe five research projects, seven development 
projects, thrbe evaluation projects, and five diffusion 
projects. Each profile contains descriptions of the structure 
and function of the project being analyzed, the specific 
outputs expected to emerge from it, the operations 
required to produce each output, and the knowledges, 
skills, and sensitivities judged to be essential to the 
performance of those operations. In addition, each profile 
contains sections dealing with the “dynamics” of project 
operations and implications that derive from the project for 
preservice staff training. The projects described range from 
small, two-man efforts within university settings to very 
large school district “projects” employing several dozen 
staff members. Eighteen of the twenty projects described 
were judged to be illustrative of the kinds of RDD&E 
activities likely to occur within the context of education in 
the future. The twenty projects account for analyses 
around 298 v^oj^ct outputs and interviews with 134 
professional staff members. 

Volume V 

A METHODOLOGY FOR THE STUDY 
OF EDUCATIONAL RDD&E 

A detailed description of the most refined form of the data 
collection methodology developed within the Studies, 
directions to guide its use, and the decision rules needed for 
the volume to function as a users manual. The volume 
includes information on procedures used in site contact, 
site preparation, data reduction and aiulyris, and i.rofile 
preparation. It also includes information on the category 
sets used in data reduction and the computerized data flies 
that contain or provide access to all data collected in the 
Studies. 



Copies of any or all of these volumes 
may be obtained at cost from 
Teaching Research 






